DeepAI AI Chat
Log In Sign Up

Adversarial attacks against Fact Extraction and VERification

by   James Thorne, et al.
University of Cambridge

This paper describes a baseline for the second iteration of the Fact Extraction and VERification shared task (FEVER2.0) which explores the resilience of systems through adversarial evaluation. We present a collection of simple adversarial attacks against systems that participated in the first FEVER shared task. FEVER modeled the assessment of truthfulness of written claims as a joint information retrieval and natural language inference task using evidence from Wikipedia. A large number of participants made use of deep neural networks in their submissions to the shared task. The extent as to whether such models understand language has been the subject of a number of recent investigations and discussion in literature. In this paper, we present a simple method of generating entailment-preserving and entailment-altering perturbations of instances by common patterns within the training data. We find that a number of systems are greatly affected with absolute losses in classification accuracy of up to 29% on the newly perturbed instances. Using these newly generated instances, we construct a sample submission for the FEVER2.0 shared task. Addressing these types of attacks will aid in building more robust fact-checking models, as well as suggest directions to expand the datasets.


page 1

page 2

page 3

page 4


The Fact Extraction and VERification (FEVER) Shared Task

We present the results of the first Fact Extraction and VERification (FE...

Generating Label Cohesive and Well-Formed Adversarial Claims

Adversarial attacks reveal important vulnerabilities and flaws of traine...

UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification

The Fact Extraction and VERification (FEVER) shared task was launched to...

DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking

The increased focus on misinformation has spurred development of data an...

Evidence-based Factual Error Correction

This paper introduces the task of factual error correction: performing e...

SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data

This paper describes the results of SemEval 2023 task 7 – Multi-Evidence...