Convex Bounds on the Softmax Function with Applications to Robustness Verification

by   Dennis Wei, et al.

The softmax function is a ubiquitous component at the output of neural networks and increasingly in intermediate layers as well. This paper provides convex lower bounds and concave upper bounds on the softmax function, which are compatible with convex optimization formulations for characterizing neural networks and other ML models. We derive bounds using both a natural exponential-reciprocal decomposition of the softmax as well as an alternative decomposition in terms of the log-sum-exp function. The new bounds are provably and/or numerically tighter than linear bounds obtained in previous work on robustness verification of transformers. As illustrations of the utility of the bounds, we apply them to verification of transformers as well as of the robustness of predictive uncertainty estimates of deep ensembles.


page 1

page 2

page 3

page 4


Robustness Verification for Transformers

Robustness verification that aims to formally certify the prediction beh...

Softmax Is Not an Artificial Trick: An Information-Theoretic View of Softmax in Neural Networks

Despite great popularity of applying softmax to map the non-normalised o...

Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers

Transformers have transformed the field of natural language processing. ...

Online normalizer calculation for softmax

The Softmax function is ubiquitous in machine learning, multiple previou...

Versatile Verification of Tree Ensembles

Machine learned models often must abide by certain requirements (e.g., f...

Lagrangian Decomposition for Neural Network Verification

A fundamental component of neural network verification is the computatio...

Enhancing Classifier Conservativeness and Robustness by Polynomiality

We illustrate the detrimental effect, such as overconfident decisions, t...

Please sign up or login with your details

Forgot password? Click here to reset