research
∙
06/30/2023
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit
In deep learning theory, the covariance matrix of the representations se...
share
research
∙
05/25/2023
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers
Autoregressive Transformers adopted in Large Language Models (LLMs) are ...
research
∙
03/16/2023
Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning
In contrast to the natural capabilities of humans to learn new tasks in ...
research
∙
10/25/2022
The Curious Case of Benign Memorization
Despite the empirical advances of deep learning across a variety of lear...
research
∙
06/07/2022
Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
Transformers have achieved remarkable success in several domains, rangin...
research
∙
05/27/2022
How Tempering Fixes Data Augmentation in Bayesian Neural Networks
While Bayesian neural networks (BNNs) provide a sound and principled alt...
research
∙
06/11/2021
Precise characterization of the prior predictive distribution of deep ReLU networks
Recent works on Bayesian neural networks (BNNs) have highlighted the nee...
research
∙
06/11/2021
Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
The "cold posterior effect" (CPE) in Bayesian deep learning describes th...
research
∙
06/29/2020