A Brief Review of Hypernetworks in Deep Learning

by   Vinod Kumar Chauhan, et al.

Hypernetworks, or hypernets in short, are neural networks that generate weights for another neural network, known as the target network. They have emerged as a powerful deep learning technique that allows for greater flexibility, adaptability, faster training, information sharing, and model compression etc. Hypernets have shown promising results in a variety of deep learning problems, including continual learning, causal inference, transfer learning, weight pruning, uncertainty quantification, zero-shot learning, natural language processing, and reinforcement learning etc. Despite their success across different problem settings, currently, there is no review available to inform the researchers about the developments and help in utilizing hypernets. To fill this gap, we review the progress in hypernets. We present an illustrative example to train deep neural networks using hypernets and propose to categorize hypernets on five criteria that affect the design of hypernets as inputs, outputs, variability of inputs and outputs, and architecture of hypernets. We also review applications of hypernets across different deep learning problem settings. Finally, we discuss the challenges and future directions that remain under-explored in the field of hypernets. We believe that hypernetworks have the potential to revolutionize the field of deep learning. They offer a new way to design and train neural networks, and they have the potential to improve the performance of deep learning models on a variety of tasks. Through this review, we aim to inspire further advancements in deep learning through hypernetworks.


page 1

page 2

page 3

page 4


Uncertainty in Natural Language Processing: Sources, Quantification, and Applications

As a main field of artificial intelligence, natural language processing ...

Deep Learning on Graphs: A Survey

Deep learning has been shown successful in a number of domains, ranging ...

Rectification-based Knowledge Retention for Continual Learning

Deep learning models suffer from catastrophic forgetting when trained in...

Zero-Shot Learning and its Applications from Autonomous Vehicles to COVID-19 Diagnosis: A Review

The challenge of learning a new concept without receiving any examples b...

Review: Deep Learning in Electron Microscopy

Deep learning is transforming most areas of science and technology, incl...

Memristive Computing for Efficient Inference on Resource Constrained Devices

The advent of deep learning has resulted in a number of applications whi...

Robust Deep Reinforcement Learning Scheduling via Weight Anchoring

Questions remain on the robustness of data-driven learning methods when ...

Please sign up or login with your details

Forgot password? Click here to reset