Towards Robustness of Neural Networks

by   Steven Basart, et al.

We introduce several new datasets namely ImageNet-A/O and ImageNet-R as well as a synthetic environment and testing suite we called CAOS. ImageNet-A/O allow researchers to focus in on the blind spots remaining in ImageNet. ImageNet-R was specifically created with the intention of tracking robust representation as the representations are no longer simply natural but include artistic, and other renditions. The CAOS suite is built off of CARLA simulator which allows for the inclusion of anomalous objects and can create reproducible synthetic environment and scenes for testing robustness. All of the datasets were created for testing robustness and measuring progress in robustness. The datasets have been used in various other works to measure their own progress in robustness and allowing for tangential progress that does not focus exclusively on natural accuracy. Given these datasets, we created several novel methods that aim to advance robustness research. We build off of simple baselines in the form of Maximum Logit, and Typicality Score as well as create a novel data augmentation method in the form of DeepAugment that improves on the aforementioned benchmarks. Maximum Logit considers the logit values instead of the values after the softmax operation, while a small change produces noticeable improvements. The Typicality Score compares the output distribution to a posterior distribution over classes. We show that this improves performance over the baseline in all but the segmentation task. Speculating that perhaps at the pixel level the semantic information of a pixel is less meaningful than that of class level information. Finally the new augmentation technique of DeepAugment utilizes neural networks to create augmentations on images that are radically different than the traditional geometric and camera based transformations used previously.


page 17

page 23

page 25

page 28

page 31

page 34

page 38

page 42


CEB Improves Model Robustness

We demonstrate that the Conditional Entropy Bottleneck (CEB) can improve...

Out of Distribution Detection on ImageNet-O

Out of distribution (OOD) detection is a crucial part of making machine ...

Does progress on ImageNet transfer to real-world datasets?

Does progress on ImageNet transfer to real-world datasets? We investigat...

Robustmix: Improving Robustness by Regularizing the Frequency Bias of Deep Nets

Deep networks have achieved impressive results on a range of well-curate...

HybridAugment++: Unified Frequency Spectra Perturbations for Model Robustness

Convolutional Neural Networks (CNN) are known to exhibit poor generaliza...

Noisy Learning for Neural ODEs Acts as a Robustness Locus Widening

We investigate the problems and challenges of evaluating the robustness ...

Please sign up or login with your details

Forgot password? Click here to reset