Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better

by   Gaurav Menghani, et al.

Deep Learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, resources required to train, etc. have all have increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code, for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. Our hope is that this survey would provide the reader with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.


page 1

page 2

page 3

page 4


Natural Language Processing Advancements By Deep Learning: A Survey

Natural Language Processing (NLP) helps empower intelligent machines by ...

Training Efficient CNNS: Tweaking the Nuts and Bolts of Neural Networks for Lighter, Faster and Robust Models

Deep Learning has revolutionized the fields of computer vision, natural ...

Optimizing Deep Learning Models For Raspberry Pi

Deep learning models have become increasingly popular for a wide range o...

A Survey on Green Deep Learning

In recent years, larger and deeper models are springing up and continuou...

Systems for Parallel and Distributed Large-Model Deep Learning Training

Deep learning (DL) has transformed applications in a variety of domains,...

On Efficient Training of Large-Scale Deep Learning Models: A Literature Review

The field of deep learning has witnessed significant progress, particula...

Sketch-based Creativity Support Tools using Deep Learning

Sketching is a natural and effective visual communication medium commonl...

Please sign up or login with your details

Forgot password? Click here to reset