Probing transfer learning with a model of synthetic correlated datasets

by   Federica Gerace, et al.

Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source task. Despite years of successful applications, transfer learning practice often relies on ad-hoc solutions, while theoretical understanding of these procedures is still limited. In the present work, we re-think a solvable model of synthetic data as a framework for modeling correlation between data-sets. This setup allows for an analytic characterization of the generalization performance obtained when transferring the learned feature map from the source to the target task. Focusing on the problem of training two-layer networks in a binary classification setting, we show that our model can capture a range of salient features of transfer learning with real data. Moreover, by exploiting parametric control over the correlation between the two data-sets, we systematically investigate under which conditions the transfer of features is beneficial for generalization.


page 8

page 17

page 21


Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks

Transfer learning has emerged as a powerful technique for improving the ...

Phase Transitions in Transfer Learning for High-Dimensional Perceptrons

Transfer learning seeks to improve the generalization performance of a t...

Transfer Learning for Endoscopic Image Classification

In this paper we propose a method for transfer learning of endoscopic im...

Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimes

Transfer learning is a useful technique for achieving improved performan...

Transferring model structure in Bayesian transfer learning for Gaussian process regression

Bayesian transfer learning (BTL) is defined in this paper as the task of...

A Correlation-Ratio Transfer Learning and Variational Stein's Paradox

A basic condition for efficient transfer learning is the similarity betw...

Evidence-empowered Transfer Learning for Alzheimer's Disease

Transfer learning has been widely utilized to mitigate the data scarcity...

Please sign up or login with your details

Forgot password? Click here to reset