Neural Processes with Stochastic Attention: Paying more attention to the context dataset

by   Mingyu Kim, et al.
KAIST 수리과학과

Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset. NPs essentially leverage a given dataset as a context representation to derive a suitable identifier for a novel task. To improve the prediction accuracy, many variants of NPs have investigated context embedding approaches that generally design novel network architectures and aggregation functions satisfying permutation invariant. In this work, we propose a stochastic attention mechanism for NPs to capture appropriate context information. From the perspective of information theory, we demonstrate that the proposed method encourages context embedding to be differentiated from a target dataset, allowing NPs to consider features in a target dataset and context embedding independently. We observe that the proposed method can appropriately capture context embedding even under noisy data sets and restricted task distributions, where typical NPs suffer from a lack of context embeddings. We empirically show that our approach substantially outperforms conventional NPs in various domains through 1D regression, predator-prey model, and image completion. Moreover, the proposed method is also validated by MovieLens-10k dataset, a real-world problem.


page 29

page 30

page 39

page 40

page 41

page 42


Reranking Passages with Coarse-to-Fine Neural Retriever using List-Context Information

Passage reranking is a crucial task in many applications, particularly w...

Latent Bottlenecked Attentive Neural Processes

Neural Processes (NPs) are popular methods in meta-learning that can est...

Dual Attention Model for Citation Recommendation

Based on an exponentially increasing number of academic articles, discov...

Recurrent Attentive Neural Process for Sequential Data

Neural processes (NPs) learn stochastic processes and predict the distri...

Fisher Information Embedding for Node and Graph Learning

Attention-based graph neural networks (GNNs), such as graph attention ne...

Embedding stochastic differential equations into neural networks via dual processes

We propose a new approach to constructing a neural network for predictin...

Constant Memory Attentive Neural Processes

Neural Processes (NPs) are efficient methods for estimating predictive u...

Please sign up or login with your details

Forgot password? Click here to reset