Volumes of logistic regression models with applications to model selection

08/05/2014
by   James G. Dowty, et al.
0

Logistic regression models with n observations and q linearly-independent covariates are shown to have Fisher information volumes which are bounded below by π^q and above by n qπ^q. This is proved with a novel generalization of the classical theorems of Pythagoras and de Gua, which is of independent interest. The finding that the volume is always finite is new, and it implies that the volume can be directly interpreted as a measure of model complexity. The volume is shown to be a continuous function of the design matrix X at generic X, but to be discontinuous in general. This means that models with sparse design matrices can be significantly less complex than nearby models, so the resulting model-selection criterion prefers sparse models. This is analogous to the way that ℓ^1-regularisation tends to prefer sparse model fits, though in our case this behaviour arises spontaneously from general principles. Lastly, an unusual topological duality is shown to exist between the ideal boundaries of the natural and expectation parameter spaces of logistic regression models.

READ FULL TEXT

page 26

page 27

research
11/09/2022

Single Parameter Inference of Non-sparse Logistic Regression Models

This paper infers a single parameter in non-sparse logistic regression m...
research
03/01/2019

On the complexity of logistic regression models

We investigate the complexity of logistic regression models which is def...
research
03/20/2012

Selection of tuning parameters in bridge regression models via Bayesian information criterion

We consider the bridge linear regression modeling, which can produce a s...
research
12/04/2019

Bayesian Group Selection in Logistic Regression with Application to MRI Data Analysis

We consider Bayesian logistic regression models with group-structured co...
research
02/28/2018

Maximum likelihood estimation of a finite mixture of logistic regression models in a continuous data stream

In marketing we are often confronted with a continuous stream of respons...
research
04/09/2018

On marginal and conditional parameters in logistic regression models

A fundamental research question is how much a variation in a covariate i...
research
01/27/2021

To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets

For finite samples with binary outcomes penalized logistic regression su...

Please sign up or login with your details

Forgot password? Click here to reset