Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimes

by   Peizhong Ju, et al.

Transfer learning is a useful technique for achieving improved performance and reducing training costs by leveraging the knowledge gained from source tasks and applying it to target tasks. Assessing the effectiveness of transfer learning relies on understanding the similarity between the ground truth of the source and target tasks. In real-world applications, tasks often exhibit partial similarity, where certain aspects are similar while others are different or irrelevant. To investigate the impact of partial similarity on transfer learning performance, we focus on a linear regression model with two distinct sets of features: a common part shared across tasks and a task-specific part. Our study explores various types of transfer learning, encompassing two options for parameter transfer. By establishing a theoretical characterization on the error of the learned model, we compare these transfer learning options, particularly examining how generalization performance changes with the number of features/parameters in both underparameterized and overparameterized regimes. Furthermore, we provide practical guidelines for determining the number of features in the common and task-specific parts for improved generalization performance. For example, when the total number of features in the source task's learning model is fixed, we show that it is more advantageous to allocate a greater number of redundant features to the task-specific part rather than the common part. Moreover, in specific scenarios, particularly those characterized by high noise levels and small true parameters, sacrificing certain true features in the common part in favor of employing more redundant features in the task-specific part can yield notable benefits.


page 1

page 2

page 3

page 4


Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks

We study the transfer learning process between two linear regression pro...

Transfer Learning Can Outperform the True Prior in Double Descent Regularization

We study a fundamental transfer learning process from source to target l...

Probing transfer learning with a model of synthetic correlated datasets

Transfer learning can significantly improve the sample efficiency of neu...

Towards All-around Knowledge Transferring: Learning From Task-irrelevant Labels

Deep neural models have hitherto achieved significant performances on nu...

Inter- and Intra-domain Knowledge Transfer for Related Tasks in Deep Character Recognition

Pre-training a deep neural network on the ImageNet dataset is a common p...

Zero Experience Required: Plug Play Modular Transfer Learning for Semantic Visual Navigation

In reinforcement learning for visual navigation, it is common to develop...

Hypothesis Transfer Learning with Surrogate Classification Losses

Hypothesis transfer learning (HTL) contrasts domain adaptation by allowi...

Please sign up or login with your details

Forgot password? Click here to reset