Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models

06/12/2023

∙

Generating intermediate steps, or Chain of Thought (CoT), is an effective way to significantly improve language models' (LM) multi-step reasoning capability. However, the CoT lengths can grow rapidly with the problem complexity, easily exceeding the maximum context size. Instead of increasing the context limit, which has already been heavily investigated, we explore an orthogonal direction: making LMs divide a problem into multiple contexts. We propose a new inference framework, called Recursion of Thought (RoT), which introduces several special tokens that the models can output to trigger context-related operations. Extensive experiments with multiple architectures including GPT-3 show that RoT dramatically improves LMs' inference capability to solve problems, whose solution consists of hundreds of thousands of tokens.

READ FULL TEXT

Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models

Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach

FedLogic: Interpretable Federated Multi-Domain Chain-of-Thought Prompt Selection for Large Language Models

Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration

Let's Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction

Rationale-Augmented Ensembles in Language Models

Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions

Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models

Related Research

Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach

FedLogic: Interpretable Federated Multi-Domain Chain-of-Thought Prompt Selection for Large Language Models

Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration

Let's Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction

Rationale-Augmented Ensembles in Language Models

Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions