RNN-Test: Adversarial Testing Framework for Recurrent Neural Network Systems

by   Jianmin Guo, et al.

While huge efforts have been investigated in the adversarial testing of convolutional neural networks (CNN), the testing for recurrent neural networks (RNN) is still limited to the classification context and leave threats for vast sequential application domains. In this work, we propose a generic adversarial testing framework RNN-Test. First, based on the distinctive structure of RNNs, we define three novel coverage metrics to measure the testing completeness and guide the generation of adversarial inputs. Second, we propose the state inconsistency orientation to generate the perturbations by maximizing the inconsistency of the hidden states of RNN cells. Finally, we combine orientations with coverage guidance to produce minute perturbations. Given the RNN model and the sequential inputs, RNN-Test will modify one character or one word out of the whole inputs based on the perturbations obtained, so as to lead the RNN to produce wrong outputs. For evaluation, we apply RNN-Test on two models of common RNN structure - the PTB language model and the spell checker model. RNN-Test efficiently reduces the performance of the PTB language model by increasing its test perplexity by 58.11 behaviors of the spell checker model with the success rate of 73.44 average. With our customization, RNN-Test using the redefined neuron coverage as guidance could achieve 35.71 DeepXplore.


Noisin: Unbiased Regularization for Recurrent Neural Networks

Recurrent neural networks (RNNs) are powerful models of sequential data....

Mandarin tone modeling using recurrent neural networks

We propose an Encoder-Classifier framework to model the Mandarin tones u...

Guiding the retraining of convolutional neural networks against adversarial inputs

Background: When using deep learning models, there are many possible vul...

Recurrent Neural Networks for Fuzz Testing Web Browsers

Generation-based fuzzing is a software testing approach which is able to...

Robustness Analysis of Classification Using Recurrent Neural Networks with Perturbed Sequential Input

For a given stable recurrent neural network (RNN) that is trained to per...

Author Identification using Multi-headed Recurrent Neural Networks

Recurrent neural networks (RNNs) are very good at modelling the flow of ...

Test Metrics for Recurrent Neural Networks

Recurrent neural networks (RNNs) have been applied to a broad range of a...

Please sign up or login with your details

Forgot password? Click here to reset