Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference

by   Tao Lei, et al.

We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation. Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase. Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge. Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approach with moderate to no accuracy loss and the same parameter efficiency.


page 2

page 9


Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models

With ever increasing parameters and computation, vision-language pre-tra...

Low-Cost Transfer Learning of Face Tasks

Do we know what the different filters of a face network represent? Can w...

Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning

Contrastive vision-language models (e.g. CLIP) are typically created by ...

Guided Transfer Learning

Machine learning requires exuberant amounts of data and computation. Als...

Learning Bound for Parameter Transfer Learning

We consider a transfer-learning problem by using the parameter transfer ...

Transfer learning for nonlinear dynamics and its application to fluid turbulence

We introduce transfer learning for nonlinear dynamics, which enables eff...

Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks

Conditional computation and modular networks have been recently proposed...

Please sign up or login with your details

Forgot password? Click here to reset