A Higher-Order Swiss Army Infinitesimal Jackknife

by   Ryan Giordano, et al.

Cross validation (CV) and the bootstrap are ubiquitous model-agnostic tools for assessing the error or variability of machine learning and statistical estimators. However, these methods require repeatedly re-fitting the model with different weighted versions of the original dataset, which can be prohibitively time-consuming. For sufficiently regular optimization problems the optimum depends smoothly on the data weights, and so the process of repeatedly re-fitting can be approximated with a Taylor series that can be often evaluated relatively quickly. The first-order approximation is known as the "infinitesimal jackknife" in the statistics literature and has been the subject of recent interest in machine learning for approximate CV. In this work, we consider high-order approximations, which we call the "higher-order infinitesimal jackknife" (HOIJ). Under mild regularity conditions, we provide a simple recursive procedure to compute approximations of all orders with finite-sample accuracy bounds. Additionally, we show that the HOIJ can be efficiently computed even in high dimensions using forward-mode automatic differentiation. We show that a linear approximation with bootstrap weights approximation is equivalent to those provided by asymptotic normal approximations. Consequently, the HOIJ opens up the possibility of enjoying higher-order accuracy properties of the bootstrap using local approximations. Consistency of the HOIJ for leave-one-out CV under different asymptotic regimes follows as corollaries from our finite-sample bounds under additional regularity assumptions. The generality of the computation and bounds motivate the name "higher-order Swiss Army infinitesimal jackknife."


page 1

page 2

page 3

page 4


Return of the Infinitesimal Jackknife

The error or variability of machine learning algorithms is often assesse...

M-estimation in GARCH models without higher order moments

We consider a class of M-estimators of the parameters of the GARCH model...

R-estimators in GARCH models; asymptotics, applications and bootstrapping

The quasi-maximum likelihood estimation is a commonly-used method for es...

Identification and estimation of Structural VARMA models using higher order dynamics

We use information from higher order moments to achieve identification o...

Higher-order asymptotics for the parametric complexity

The parametric complexity is the key quantity in the minimum description...

Higher-Order Correct Multiplier Bootstraps for Count Functionals of Networks

Subgraph counts play a central role in both graph limit theory and netwo...

New Edgeworth-type expansions with finite sample guarantees

We establish Edgeworth-type expansions for a difference between probabil...

Please sign up or login with your details

Forgot password? Click here to reset