Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills

by   Ori Yoran, et al.
Tel Aviv University

Models pre-trained with a language modeling objective possess ample world knowledge and language skills, but are known to struggle in tasks that require reasoning. In this work, we propose to leverage semi-structured tables, and automatically generate at scale question-paragraph pairs, where answering the question requires reasoning over multiple facts in the paragraph. We add a pre-training step over this synthetic data, which includes examples that require 16 different reasoning skills such as number comparison, conjunction, and fact composition. To improve data efficiency, we propose sampling strategies that focus training on reasoning skills the model is currently lacking. We evaluate our approach on three reading comprehension datasets that are focused on reasoning, and show that our model, PReasM, substantially outperforms T5, a popular pre-trained encoder-decoder model. Moreover, sampling examples based on current model errors leads to faster training and higher overall performance.


Injecting Numerical Reasoning Skills into Language Models

Large pre-trained language models (LMs) are known to encode substantial ...

ALERT: Adapting Language Models to Reasoning Tasks

Current large language models can perform reasonably well on complex tas...

Comprehension Based Question Answering using Bloom's Taxonomy

Current pre-trained language models have lots of knowledge, but a more l...

Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge

To what extent can a neural network systematically reason over symbolic ...

ReasonBERT: Pre-trained to Reason with Distant Supervision

We present ReasonBert, a pre-training method that augments language mode...

NT5?! Training T5 to Perform Numerical Reasoning

Numerical reasoning over text (NRoT) presents unique challenges that are...

Overcoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmetic

Through their transfer learning abilities, highly-parameterized large pr...

Please sign up or login with your details

Forgot password? Click here to reset