research
∙
09/17/2023
Contrastive Decoding Improves Reasoning in Large Language Models
We demonstrate that Contrastive Decoding – a simple, computationally lig...
share
research
∙
08/11/2023
Self-Alignment with Instruction Backtranslation
We present a scalable method to build a high quality instruction followi...
research
∙
05/24/2023
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
Language models (LMs) often struggle to pay enough attention to the inpu...
research
∙
05/23/2023
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Evaluating the factuality of long-form text generated by large language ...
research
∙
05/18/2023
LIMA: Less Is More for Alignment
Large language models are trained in two stages: (1) unsupervised pretra...
research
∙
05/12/2023
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Autoregressive transformers are spectacular models for short sequences b...
research
∙
05/06/2023
Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization
Prompt tuning is one of the successful approaches for parameter-efficien...
research
∙
03/24/2023
Scaling Expert Language Models with Unsupervised Domain Discovery
Large language models are typically trained densely: all parameters are ...
research
∙
01/30/2023
REPLUG: Retrieval-Augmented Black-Box Language Models
We introduce REPLUG, a retrieval-augmented language modeling framework t...
research
∙
01/29/2023
Progressive Prompts: Continual Learning for Language Models
We introduce Progressive Prompts - a simple and efficient approach for c...
research
∙
12/15/2022
Improving Chess Commentaries by Combining Language Models with Symbolic Reasoning Engines
Despite many recent advancements in language modeling, state-of-the-art ...
research
∙
12/05/2022
In-context Examples Selection for Machine Translation
Large-scale generative models show an impressive ability to perform a wi...
research
∙
12/02/2022
Nonparametric Masked Language Modeling
Existing language models (LMs) predict tokens with a softmax over a fini...
research
∙
11/29/2022
Coder Reviewer Reranking for Code Generation
Sampling diverse programs from a code language model and reranking with ...
research
∙
11/22/2022
AutoReply: Detecting Nonsense in Dialogue Introspectively with Discriminative Replies
Existing approaches built separate classifiers to detect nonsense in dia...
research
∙
11/22/2022
Retrieval-Augmented Multimodal Language Modeling
Recent multimodal models such as DALL-E and CM3 have achieved remarkable...
research
∙
10/27/2022
Contrastive Decoding: Open-ended Text Generation as Optimization
Likelihood, although useful as a training loss, is a poor search objecti...
research
∙
10/07/2022
Measuring and Narrowing the Compositionality Gap in Language Models
We investigate the ability of language models to perform compositional r...
research
∙
08/15/2022
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Large language models have been widely adopted but require significant G...
research
∙
08/05/2022
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
We present Branch-Train-Merge (BTM), a communication-efficient algorithm...
research
∙
06/21/2022
Questions Are All You Need to Train a Dense Passage Retriever
We introduce ART, a new corpus-level autoencoding approach for training ...
research
∙
06/07/2022
LegoNN: Building Modular Encoder-Decoder Models
State-of-the-art encoder-decoder models (e.g. for machine translation (M...
research
∙
05/09/2022
Few-shot Mining of Naturally Occurring Inputs and Outputs
Creating labeled natural language training data is expensive and require...
research
∙
04/15/2022
Improving Passage Retrieval with Zero-Shot Question Generation
We propose a simple and effective re-ranking method for improving passag...
research
∙
04/12/2022
InCoder: A Generative Model for Code Infilling and Synthesis
Code is seldom written in a single left-to-right pass and is instead rep...
research
∙
02/25/2022
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Large language models (LMs) are able to in-context learn – perform a new...
research
∙
01/19/2022
CM3: A Causal Masked Multimodal Model of the Internet
We introduce CM3, a family of causally masked generative models trained ...
research
∙
10/29/2021
MetaICL: Learning to Learn In Context
We introduce MetaICL (Meta-training for In-Context Learning), a new meta...
research
∙
10/16/2021
Sparse Distillation: Speeding Up Text Classification by Using Bigger Models
Distilling state-of-the-art transformer models into lightweight student ...
research
∙
10/15/2021
Tricks for Training Sparse Translation Models
Multi-task learning with an unbalanced data distribution skews model lea...
research
∙
10/06/2021
8-bit Optimizers via Block-wise Quantization
Stateful optimizers maintain gradient statistics over time, e.g., the ex...
research
∙
08/27/2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Since the introduction of the transformer model by Vaswani et al. (2017)...
research
∙
08/11/2021
DEMix Layers: Disentangling Domains for Modular Language Modeling
We introduce a new domain expert mixture (DEMix) layer that enables cond...
research
∙
08/09/2021
Noisy Channel Language Model Prompting for Few-Shot Text Classification
We introduce a noisy channel approach for language model prompting in fe...
research
∙
07/14/2021
HTLM: Hyper-Text Pre-Training and Prompting of Language Models
We introduce HTLM, a hyper-text language model trained on a large-scale ...
research
∙
06/15/2021
Question Answering Infused Pre-training of General-Purpose Contextualized Representations
This paper proposes a pre-training objective based on question answering...
research
∙
04/15/2021
Multitasking Inhibits Semantic Drift
When intelligent agents communicate to accomplish shared goals, how do t...
research
∙
03/30/2021
BASE Layers: Simplifying Training of Large, Sparse Models
We introduce a new balanced assignment of experts (BASE) layer for large...
research
∙
12/31/2020
Shortformer: Better Language Modeling using Shorter Inputs
We explore the benefits of decreasing the input length of transformers. ...
research
∙
12/30/2020
Joint Verification and Reranking for Open Fact Checking Over Tables
Structured information is an important knowledge source for automatic ve...
research
∙
10/01/2020
Nearest Neighbor Machine Translation
We introduce k-nearest-neighbor machine translation (kNN-MT), which pred...
research
∙
09/28/2020
Conversational Semantic Parsing
The structured representation for semantic parsing in task-oriented assi...
research
∙
09/16/2020
Grounded Adaptation for Zero-shot Executable Semantic Parsing
We propose Grounded Adaptation for Zero-shot Executable Semantic Parsing...
research
∙
06/26/2020
Pre-training via Paraphrasing
We introduce MARGE, a pre-trained sequence-to-sequence model learned wit...
research
∙
05/22/2020
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Large pre-trained language models have been shown to store factual knowl...
research
∙
04/08/2020
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Practical applications of abstractive summarization models are limited b...
research
∙
01/22/2020
Multilingual Denoising Pre-training for Neural Machine Translation
This paper demonstrates that multilingual denoising pre-training produce...
research
∙
11/09/2019
Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
Inspired by modular software design principles of independence, intercha...
research
∙
11/01/2019
Generalization through Memorization: Nearest Neighbor Language Models
We introduce kNN-LMs, which extend a pre-trained neural language model (...
research
∙
10/29/2019