Acceptability Judgements via Examining the Topology of Attention Maps

by   Daniil Cherniavskii, et al.

The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the ability of the attention heads to judge the grammatical acceptability of a sentence has been underexplored. This paper approaches the paradigm of acceptability judgments with topological data analysis (TDA), showing that the geometric properties of the attention graph can be efficiently exploited for two standard practices in linguistics: binary judgments and linguistic minimal pairs. Topological features enhance the BERT-based acceptability classifier scores by 8 languages (English, Italian, and Swedish). By revealing the topological discrepancy between attention maps of minimal pairs, we achieve the human-level performance on the BLiMP benchmark, outperforming nine statistical and Transformer LM baselines. At the same time, TDA provides the foundation for analyzing the linguistic functions of attention heads and interpreting the correspondence between the graph features and grammatical phenomena.


Can BERT eat RuCoLA? Topological Data Analysis to Explain

This paper investigates how Transformer language models (LMs) fine-tuned...

The Topological BERT: Transforming Attention into Topology for Natural Language Processing

In recent years, the introduction of the Transformer models sparked a re...

BLiMP: A Benchmark of Linguistic Minimal Pairs for English

We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLi...

Topological Data Analysis for Speech Processing

We apply topological data analysis (TDA) to speech classification proble...

PAT: Parallel Attention Transformer for Visual Question Answering in Vietnamese

We present in this paper a novel scheme for multimodal learning named th...

Betti numbers of attention graphs is all you really need

We apply methods of topological analysis to the attention graphs, calcul...

Harnessing the richness of the linguistic signal in predicting pragmatic inferences

The strength of pragmatic inferences systematically depends on linguisti...

Please sign up or login with your details

Forgot password? Click here to reset