Lower Bounds on Cross-Entropy Loss in the Presence of Test-time Adversaries

by   Arjun Nitin Bhagoji, et al.

Understanding the fundamental limits of robust supervised learning has emerged as a problem of immense interest, from both practical and theoretical standpoints. In particular, it is critical to determine classifier-agnostic bounds on the training loss to establish when learning is possible. In this paper, we determine optimal lower bounds on the cross-entropy loss in the presence of test-time adversaries, along with the corresponding optimal classification outputs. Our formulation of the bound as a solution to an optimization problem is general enough to encompass any loss function depending on soft classifier outputs. We also propose and provide a proof of correctness for a bespoke algorithm to compute this lower bound efficiently, allowing us to determine lower bounds for multiple practical datasets of interest. We use our lower bounds as a diagnostic tool to determine the effectiveness of current robust training methods and find a gap from optimality at larger budgets. Finally, we investigate the possibility of using of optimal classification outputs as soft labels to empirically improve robust training.


page 1

page 2

page 3

page 4


Characterizing the Optimal 0-1 Loss for Multi-class Classification with a Test-time Attacker

Finding classifiers robust to adversarial examples is critical for their...

Lower Bounds on Adversarial Robustness from Optimal Transport

While progress has been made in understanding the robustness of machine ...

Lower bounds for prams over Z

This paper presents a new abstract method for proving lower bounds in co...

Signal to Noise Ratio Loss Function

This work proposes a new loss function targeting classification problems...

Test-Time Adaptation via Conjugate Pseudo-labels

Test-time adaptation (TTA) refers to adapting neural networks to distrib...

Language-Aware Soft Prompting for Vision Language Foundation Models

This paper is on soft prompt learning for Vision & Language (V L) mode...

Loss Bounds for Approximate Influence-Based Abstraction

Sequential decision making techniques hold great promise to improve the ...

Please sign up or login with your details

Forgot password? Click here to reset