research
∙
10/27/2022
What Language Model to Train if You Have One Million GPU Hours?
The crystallization of modeling methods around the Transformer architect...
share
research
∙
10/07/2022
Measuring and Narrowing the Compositionality Gap in Language Models
We investigate the ability of language models to perform compositional r...
research
∙
03/30/2022
Transformer Language Models without Positional Encodings Still Learn Positional Information
Transformers typically require some form of positional encoding, such as...
research
∙
08/27/2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Since the introduction of the transformer model by Vaswani et al. (2017)...
research
∙
12/31/2020
Shortformer: Better Language Modeling using Shorter Inputs
We explore the benefits of decreasing the input length of transformers. ...
research
∙
11/10/2019
Improving Transformer Models by Reordering their Sublayers
Multilayer transformer networks consist of interleaved self-attention an...
research
∙
03/11/2019
Partially Shuffling the Training Data to Improve Language Models
Although SGD requires shuffling the training data between epochs, curren...
research
∙
10/31/2018
You May Not Need Attention
In NMT, how far can we get without attention and without separate encodi...
research
∙
06/05/2017
Language Generation with Recurrent Generative Adversarial Networks without Pre-training
Generative Adversarial Networks (GANs) have shown great promise recently...
research
∙
08/20/2016