Understanding tables with intermediate pre-training

Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning. While there is extensive work on textual entailment, table entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task. To be able to use long examples as input of BERT models, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a moderate drop in accuracy. The different methods set the new state-of-the-art on the TabFact (Chen et al., 2020) and SQA datasets.


page 1

page 2

page 3

page 4


TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training

We present the TAPAS contribution to the Shared Task on Statement Verifi...

Table-To-Text generation and pre-training with TabT5

Encoder-only transformer models have been successfully applied to differ...

Multilingual Constituency Parsing with Self-Attention and Pre-Training

We extend our previous work on constituency parsing (Kitaev and Klein, 2...

Dr.Quad at MEDIQA 2019: Towards Textual Inference and Question Entailment using contextualized representations

This paper presents the submissions by Team Dr.Quad to the ACL-BioNLP 20...

MATE: Multi-view Attention for Table Transformer Efficiency

This work presents a sparse-attention Transformer architecture for model...

Language Representation Models for Fine-Grained Sentiment Classification

Sentiment classification is a quickly advancing field of study with appl...

ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns' Semantic Properties and their Prototypicality

Large scale language models encode rich commonsense knowledge acquired t...

Please sign up or login with your details

Forgot password? Click here to reset