DeepAI AI Chat
Log In Sign Up

ANTONIO: Towards a Systematic Method of Generating NLP Benchmarks for Verification

by   Marco Casadio, et al.
University of Birmingham
Hebrew University of Jerusalem
Heriot-Watt University

Verification of machine learning models used in Natural Language Processing (NLP) is known to be a hard problem. In particular, many known neural network verification methods that work for computer vision and other numeric datasets do not work for NLP. Here, we study technical reasons that underlie this problem. Based on this analysis, we propose practical methods and heuristics for preparing NLP datasets and models in a way that renders them amenable to known verification methods based on abstract interpretation. We implement these methods as a Python library called ANTONIO that links to the neural network verifiers ERAN and Marabou. We perform evaluation of the tool using an NLP dataset R-U-A-Robot suggested as a benchmark for verifying legally critical NLP applications. We hope that, thanks to its general applicability, this work will open novel possibilities for including NLP verification problems into neural network verification competitions, and will popularise NLP problems within this community.


Deep Learning applied to NLP

Convolutional Neural Network (CNNs) are typically associated with Comput...

Spark NLP: Natural Language Understanding at Scale

Spark NLP is a Natural Language Processing (NLP) library built on top of...

A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing

Many natural language processing (NLP) tasks are naturally imbalanced, a...

SqueezeBERT: What can computer vision teach NLP about efficient neural networks?

Humans read and write hundreds of billions of messages every day. Furthe...

A Neural Few-Shot Text Classification Reality Check

Modern classification models tend to struggle when the amount of annotat...

FewJoint: A Few-shot Learning Benchmark for Joint Language Understanding

Few-learn learning (FSL) is one of the key future steps in machine learn...

Human Interpretable AI: Enhancing Tsetlin Machine Stochasticity with Drop Clause

In this article, we introduce a novel variant of the Tsetlin machine (TM...