Calibrated Model Criticism Using Split Predictive Checks

by   Jiawei Li, et al.

Checking how well a fitted model explains the data is one of the most fundamental parts of a Bayesian data analysis. However, existing model checking methods suffer from trade-offs between being well-calibrated, automated, and computationally efficient. To overcome these limitations, we propose split predictive checks (SPCs), which combine the ease-of-use and speed of posterior predictive checks with the good calibration properties of predictive checks that rely on model-specific derivations or inference schemes. We develop an asymptotic theory for two types of SPCs: single SPCs and the divided SPC. Our results demonstrate that they offer complementary strengths: single SPCs provide superior power in the small-data regime or when the misspecification is significant and divided SPCs provide superior power as the dataset size increases or when the form of misspecification is more subtle. We validate the finite-sample utility of SPCs through extensive simulation experiments in exponential family and hierarchical models, and provide four real-data examples where SPCs offer novel insights and additional flexibility beyond what is available when using posterior predictive checks.


Conformal Bayesian Computation

We develop scalable methods for producing conformal Bayesian predictive ...

Conformal calibrators

Most existing examples of full conformal predictive systems, split-confo...

Bayesian model checking: A comparison of tests

Two procedures for checking Bayesian models are compared using a simple ...

Statistical Assessment of Replicability via Bayesian Model Criticism

Assessment of replicability is critical to ensure the quality and rigor ...

Population Predictive Checks

Bayesian modeling has become a staple for researchers analyzing data. Th...

Computational methods for fast Bayesian model assessment via calibrated posterior p-values

Posterior predictive p-values (ppps) have become popular tools for Bayes...

Posterior Dispersion Indices

Probabilistic modeling is cyclical: we specify a model, infer its poster...

Please sign up or login with your details

Forgot password? Click here to reset