Efficient estimation of the ANOVA mean dimension, with an application to neural net classification

by   Christopher Hoyt, et al.

The mean dimension of a black box function of d variables is a convenient way to summarize the extent to which it is dominated by high or low order interactions. It is expressed in terms of 2^d-1 variance components but it can be written as the sum of d Sobol' indices that can be estimated by leave one out methods. We compare the variance of these leave one out methods: a Gibbs sampler called winding stairs, a radial sampler that changes each variable one at a time from a baseline, and a naive sampler that never reuses function evaluations and so costs about double the other methods. For an additive function the radial and winding stairs are most efficient. For a multiplicative function the naive method can easily be most efficient if the factors have high kurtosis. As an illustration we consider the mean dimension of a neural network classifier of digits from the MNIST data set. The classifier is a function of 784 pixels. For that problem, winding stairs is the best algorithm. We find that inputs to the final softmax layer have mean dimensions ranging from 1.35 to 2.0.


page 13

page 17


Mean dimension of radial basis functions

We show that generalized multiquadric radial basis functions (RBFs) on ℝ...

Variable-Complexity Weighted-Tempered Gibbs Samplers for Bayesian Variable Selection

Subset weighted-Tempered Gibbs Sampler (wTGS) has been recently introduc...

Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis

We describe a novel attribution method which is grounded in Sensitivity ...

Accelerating Derivative-Free Optimization with Dimension Reduction and Hyperparameter Learning

We consider convex, black-box objective functions with additive or multi...

Deletion and Insertion Tests in Regression Models

A basic task in explainable AI (XAI) is to identify the most important f...

The Mean Dimension of Neural Networks – What causes the interaction effects?

Owen and Hoyt recently showed that the effective dimension offers key st...

Please sign up or login with your details

Forgot password? Click here to reset