research
∙
09/11/2023
Uncovering mesa-optimization algorithms in Transformers
Transformers have become the dominant model in deep learning, but the re...
share
research
∙
09/04/2023
Gated recurrent neural networks discover attention
Recent architectural developments have enabled recurrent neural networks...
research
∙
12/15/2022
Transformers learn in-context by gradient descent
Transformers have become the state-of-the-art neural network architectur...
research
∙
10/18/2022
Disentangling the Predictive Variance of Deep Ensembles through the Neural Tangent Kernel
Identifying unfamiliar inputs, also known as out-of-distribution (OOD) d...
research
∙
09/15/2022
Random initialisations performing above chance and how to find them
Neural networks trained with stochastic gradient descent (SGD) starting ...
research
∙
07/04/2022
The least-control principle for learning at equilibrium
Equilibrium systems are a powerful way to express neural computations. A...
research
∙
10/27/2021
Learning where to learn: Gradient sparsity in meta and continual learning
Finding neural network weights that generalize well from small datasets ...
research
∙
04/04/2021
A contrastive rule for meta-learning
Meta-learning algorithms leverage regularities that are present on a set...
research
∙
03/01/2021
Posterior Meta-Replay for Continual Learning
Continual Learning (CL) algorithms have recently received a lot of atten...
research
∙
07/25/2020
Economical ensembles with hypernetworks
Averaging the predictions of many independently trained neural networks ...
research
∙
06/22/2020
Continual Learning in Recurrent Neural Networks with Hypernetworks
The last decade has seen a surge of interest in continual learning (CL),...
research
∙
06/03/2019