Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

10/04/2021

∙

Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

READ FULL TEXT

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

Uncovering More Shallow Heuristics: Probing the Natural Language Inference Capacities of Transformer-Based Pre-Trained Language Models Using Syllogistic Patterns

Deepfake Detection: A Comparative Analysis

Transformer Models for Type Inference in the Simply Typed Lambda Calculus: A Case Study in Deep Learning for Code

When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks

The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Quantity doesn't buy quality syntax with neural language models

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Related Research

Compressing Large-Scale Transformer-Based Models: A Case Study on BERT

Uncovering More Shallow Heuristics: Probing the Natural Language Inference Capacities of Transformer-Based Pre-Trained Language Models Using Syllogistic Patterns

Deepfake Detection: A Comparative Analysis

Transformer Models for Type Inference in the Simply Typed Lambda Calculus: A Case Study in Deep Learning for Code

When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks

The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Quantity doesn't buy quality syntax with neural language models