Virtual Adversarial Training: a Regularization Method for Supervised and Semi-supervised Learning

by   Takeru Miyato, et al.

We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the output distribution. Virtual adversarial loss is defined as the robustness of the model's posterior distribution against local perturbation around each input data point. Our method is similar to adversarial training, but differs from adversarial training in that it determines the adversarial direction based only on the output distribution and that it is applicable to a semi-supervised setting. Because the directions in which we smooth the model are virtually adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward and backpropagations. In our experiments, we applied VAT to supervised and semi-supervised learning on multiple benchmark datasets. With additional improvement based on entropy minimization principle, our VAT achieves the state-of-the-art performance on SVHN and CIFAR-10 for semi-supervised learning tasks.


page 9

page 11


Distributional Smoothing with Virtual Adversarial Training

We propose local distributional smoothness (LDS), a new notion of smooth...

Adversarial Transformations for Semi-Supervised Learning

We propose a Regularization framework based on Adversarial Transformatio...

Adversarial Training Methods for Semi-Supervised Text Classification

Adversarial training provides a means of regularizing supervised learnin...

Negative sampling in semi-supervised learning

We introduce Negative Sampling in Semi-Supervised Learning (NS3L), a sim...

Understanding and Improving Virtual Adversarial Training

In semi-supervised learning, virtual adversarial training (VAT) approach...

Tangent-Normal Adversarial Regularization for Semi-supervised Learning

The ever-increasing size of modern datasets combined with the difficulty...

Regularization with Latent Space Virtual Adversarial Training

Virtual Adversarial Training (VAT) has shown impressive results among re...

Code Repositories


Virtual adversarial training with Tensorflow

view repo


Implementation of Virtual adversarial training with chainer

view repo


Virtual Adversarial Training on TensorFlow

view repo

Please sign up or login with your details

Forgot password? Click here to reset