Transfer learning of regression models from a sequence of datasets by penalized estimation

by   Wessel N. van Wieringen, et al.

Transfer learning refers to the promising idea of initializing model fits based on pre-training on other data. We particularly consider regression modeling settings where parameter estimates from previous data can be used as anchoring points, yet may not be available for all parameters, thus covariance information cannot be reused. A procedure that updates through targeted penalized estimation, which shrinks the estimator towards a nonzero value, is presented. The parameter estimate from the previous data serves as this nonzero value when an update is sought from novel data. This naturally extends to a sequence of data sets with the same response, but potentially only partial overlap in covariates. The iteratively updated regression parameter estimator is shown to be asymptotically unbiased and consistent. The penalty parameter is chosen through constrained cross-validated loglikelihood optimization. The constraint bounds the amount of shrinkage of the updated estimator toward the current one from below. The bound aims to preserve the (updated) estimator's goodness-of-fit on all-but-the-novel data. The proposed approach is compared to other regression modeling procedures. Finally, it is illustrated on an epidemiological study where the data arrive in batches with different covariate-availability and the model is re-fitted with the availability of a novel batch.


Jeffreys' prior, finiteness and shrinkage in binomial-response generalized linear models

This paper studies the finiteness properties of a reduced-bias estimator...

Transfer learning with high-dimensional quantile regression

Transfer learning has become an essential technique to exploit informati...

On the existence of Firth's modified estimates in logistic regression models

In logistic regression modeling, Firth's modified estimator is widely us...

Multi-Model Penalized Regression

Model fitting often aims to fit a single model, assuming that the impose...

Nonlinear regression: finite sample guarantees

This paper offers a new approach for study the frequentist properties of...

Targeted Fused Ridge Estimation of Inverse Covariance Matrices from Multiple High-Dimensional Data Classes

We consider the problem of jointly estimating multiple precision matrice...

Generation of new exciting regressors for consistent on-line estimation for a scalar parameter

In this paper the problem of estimation of a single parameter from a lin...

Please sign up or login with your details

Forgot password? Click here to reset