DeepAI AI Chat
Log In Sign Up

Randomized Smoothing with Masked Inference for Adversarially Robust Text Classifications

by   Han Cheol Moon, et al.

Large-scale pre-trained language models have shown outstanding performance in a variety of NLP tasks. However, they are also known to be significantly brittle against specifically crafted adversarial examples, leading to increasing interest in probing the adversarial robustness of NLP systems. We introduce RSMI, a novel two-stage framework that combines randomized smoothing (RS) with masked inference (MI) to improve the adversarial robustness of NLP systems. RS transforms a classifier into a smoothed classifier to obtain robust representations, whereas MI forces a model to exploit the surrounding context of a masked token in an input sequence. RSMI improves adversarial robustness by 2 to 3 times over existing state-of-the-art methods on benchmark datasets. We also perform in-depth qualitative analysis to validate the effectiveness of the different stages of RSMI and probe the impact of its components through extensive ablations. By empirically proving the stability of RSMI, we put it forward as a practical method to robustly train large-scale NLP models. Our code and datasets are available at


page 1

page 2

page 3

page 4


In and Out-of-Domain Text Adversarial Robustness via Label Smoothing

Recently it has been shown that state-of-the-art NLP models are vulnerab...

BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese

We present BARTpho with two versions – BARTpho_word and BARTpho_syllable...

Robust and Accurate – Compositional Architectures for Randomized Smoothing

Randomized Smoothing (RS) is considered the state-of-the-art approach to...

Visually-augmented pretrained language models for NLP tasks without images

Although pre-trained language models (PLMs) have shown impressive perfor...

Second-Order NLP Adversarial Examples

Adversarial example generation methods in NLP rely on models like langua...

Boosting Randomized Smoothing with Variance Reduced Classifiers

Randomized Smoothing (RS) is a promising method for obtaining robustness...

Image Synthesis with a Single (Robust) Classifier

We show that the basic classification framework alone can be used to tac...