Optimal transfer protocol by incremental layer defrosting

by   Federica Gerace, et al.

Transfer learning is a powerful tool enabling model training with limited amounts of data. This technique is particularly useful in real-world problems where data availability is often a serious limitation. The simplest transfer learning protocol is based on “freezing" the feature-extractor layers of a network pre-trained on a data-rich source task, and then adapting only the last layers to a data-poor target task. This workflow is based on the assumption that the feature maps of the pre-trained model are qualitatively similar to the ones that would have been learned with enough data on the target task. In this work, we show that this protocol is often sub-optimal, and the largest performance gain may be achieved when smaller portions of the pre-trained network are kept frozen. In particular, we make use of a controlled framework to identify the optimal transfer depth, which turns out to depend non-trivially on the amount of available training data and on the degree of source-target task correlation. We then characterize transfer optimality by analyzing the internal representations of two networks trained from scratch on the source and the target task through multiple established similarity measures.


page 1

page 2

page 3

page 4


Disposable Transfer Learning for Selective Source Task Unlearning

Transfer learning is widely used for training deep neural networks (DNN)...

Adversarially-Trained Deep Nets Transfer Better

Transfer learning has emerged as a powerful methodology for adapting pre...

Transfer Learning with Human Corneal Tissues: An Analysis of Optimal Cut-Off Layer

Transfer learning is a powerful tool to adapt trained neural networks to...

TransferD2: Automated Defect Detection Approach in Smart Manufacturing using Transfer Learning Techniques

Quality assurance is crucial in the smart manufacturing industry as it i...

Fast and Accurate Transferability Measurement by Evaluating Intra-class Feature Variance

Given a set of pre-trained models, how can we quickly and accurately fin...

EXPANSE: A Deep Continual / Progressive Learning System for Deep Transfer Learning

Deep transfer learning techniques try to tackle the limitations of deep ...

PAC-Net: A Model Pruning Approach to Inductive Transfer Learning

Inductive transfer learning aims to learn from a small amount of trainin...

Please sign up or login with your details

Forgot password? Click here to reset