DeepAI AI Chat
Log In Sign Up

Meta-Learning with Self-Improving Momentum Target

by   Jihoon Tack, et al.
KAIST 수리과학과

The idea of using a separately trained target model (or teacher) to improve the performance of the student model has been increasingly popular in various machine learning domains, and meta-learning is no exception; a recent discovery shows that utilizing task-wise target models can significantly boost the generalization performance. However, obtaining a target model for each task can be highly expensive, especially when the number of tasks for meta-learning is large. To tackle this issue, we propose a simple yet effective method, coined Self-improving Momentum Target (SiMT). SiMT generates the target model by adapting from the temporal ensemble of the meta-learner, i.e., the momentum network. This momentum network and its task-specific adaptations enjoy a favorable generalization performance, enabling self-improving of the meta-learner through knowledge distillation. Moreover, we found that perturbing parameters of the meta-learner, e.g., dropout, further stabilize this self-improving process by preventing fast convergence of the distillation loss during meta-training. Our experimental results demonstrate that SiMT brings a significant performance gain when combined with a wide range of meta-learning methods under various applications, including few-shot regression, few-shot classification, and meta-reinforcement learning. Code is available at


page 1

page 2

page 3

page 4


Meta Learning for Knowledge Distillation

We present Meta Learning for Knowledge Distillation (MetaDistil), a simp...

Few-Shot Learning of Compact Models via Task-Specific Meta Distillation

We consider a new problem of few-shot learning of compact models. Meta-l...

Exploring Effective Factors for Improving Visual In-Context Learning

The In-Context Learning (ICL) is to understand a new task via a few demo...

Collaborative Distillation Meta Learning for Simulation Intensive Hardware Design

This paper proposes a novel collaborative distillation meta learning (CD...

Efficient time stepping for numerical integration using reinforcement learning

Many problems in science and engineering require the efficient numerical...

VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution

Since the introduction of deep learning, a wide scope of representation ...

Meta Learning with Differentiable Closed-form Solver for Fast Video Object Segmentation

This paper tackles the problem of video object segmentation. We are spec...

Code Repositories


Meta-Learning with Self-Improving Momentum Target (NeurIPS 2022)

view repo