Significance tests of feature relevance for a blackbox learner

by   Ben Dai, et al.

An exciting recent development is the uptake of deep learning in many scientific fields, where the objective is seeking novel scientific insights and discoveries. To interpret a learning outcome, researchers perform hypothesis testing for explainable features to advance scientific domain knowledge. In such a situation, testing for a blackbox learner poses a severe challenge because of intractable models, unknown limiting distributions of parameter estimates, and high computational constraints. In this article, we derive two consistent tests for the feature relevance of a blackbox learner. The first one evaluates a loss difference with perturbation on an inference sample, which is independent of an estimation sample used for parameter estimation in model fitting. The second further splits the inference sample into two but does not require data perturbation. Also, we develop their combined versions by aggregating the order statistics of the p-values based on repeated sample splitting. To estimate the splitting ratio and the perturbation size, we develop adaptive splitting schemes for suitably controlling the Type 1 error subject to computational constraints. By deflating the bias-sd-ratio, we establish asymptotic null distributions of the test statistics and their consistency in terms of statistical power. Our theoretical power analysis and simulations indicate that the one-split test is more powerful than the two-split test, though the latter is easier to apply for large datasets. Moreover, the combined tests are more stable while compensating for a power loss by repeated sample splitting. Numerically, we demonstrate the utility of the proposed tests on two benchmark examples. Accompanying this paper is our Python library dnn-inference that implements the proposed tests.


page 22

page 24


Learning Kernel Tests Without Data Splitting

Modern large-scale kernel-based tests such as maximum mean discrepancy (...

On the choice of the splitting ratio for the split likelihood ratio test

The recently introduced framework of universal inference provides a new ...

Finite Sample Hypothesis Tests for Stacked Estimating Equations

Suppose there are two unknown parameters, each parameter is the solution...

Testing Mediation Effects Using Logic of Boolean Matrices

Mediation analysis is becoming an increasingly important tool in scienti...

A stable and adaptive polygenic signal detection method based on repeated sample splitting

Using polygenic risk score for trait association analyses and disease pr...

Multiple-Splitting Projection Test for High-Dimensional Mean Vectors

We propose a multiple-splitting projection test (MPT) for one-sample mea...

The power of private likelihood-ratio tests for goodness-of-fit in frequency tables

Privacy-protecting data analysis investigates statistical methods under ...

Please sign up or login with your details

Forgot password? Click here to reset