research
∙
08/08/2023
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
The legality of training language models (LMs) on copyrighted or otherwi...
share
research
∙
07/19/2023
Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
Rising computational demands of modern natural language processing (NLP)...
research
∙
07/13/2023
DIALGEN: Collaborative Human-LM Generated Dialogues for Improved Understanding of Human-Human Conversations
Applications that could benefit from automatic understanding of human-hu...
research
∙
06/24/2023
Estimating the Causal Effect of Early ArXiving on Paper Acceptance
What is the effect of releasing a preprint of a paper before it is submi...
research
∙
06/16/2023
Reproducibility in NLP: What Have We Learned from the Checklist?
Scientific progress in NLP rests on the reproducibility of researchers' ...
research
∙
06/09/2023
Morphosyntactic probing of multilingual BERT models
We introduce an extensive dataset for multilingual probing of morphologi...
research
∙
06/03/2023
Stubborn Lexical Bias in Data and Models
In NLP, recent work has seen increased focus on spurious correlations be...
research
∙
06/02/2023
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
Language models (LMs) often exhibit undesirable text generation behavior...
research
∙
03/24/2023
Scaling Expert Language Models with Unsupervised Domain Discovery
Large language models are typically trained densely: all parameters are ...
research
∙
01/11/2023
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference
Large-scale language model pretraining is a very successful form of self...
research
∙
12/08/2022
Demystifying Prompts in Language Models via Perplexity Estimation
Language models can be prompted to perform a wide variety of zero- and f...
research
∙
12/01/2022
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors
Language models trained on massive prompted multitask datasets like T0 (...
research
∙
11/30/2022
Domain Mismatch Doesn't Always Prevent Cross-Lingual Transfer Learning
Cross-lingual transfer learning without labeled target language data or ...
research
∙
11/15/2022
PromptCap: Prompt-Guided Task-Aware Image Captioning
Image captioning aims to describe an image with a natural language sente...
research
∙
11/07/2022
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers
The attention mechanism is considered the backbone of the widely-used Tr...
research
∙
10/16/2022
Modeling Context With Linear Attention for Scalable Document-Level Translation
Document-level machine translation leverages inter-sentence dependencies...
research
∙
10/14/2022
Transparency Helps Reveal When Language Models Learn Meaning
Many current NLP systems are built from language models trained to optim...
research
∙
10/07/2022
Measuring and Narrowing the Compositionality Gap in Language Models
We investigate the ability of language models to perform compositional r...
research
∙
10/06/2022
Binding Language Models in Symbolic Languages
Though end-to-end neural approaches have recently been dominating NLP ta...
research
∙
09/02/2022
Elaboration-Generating Commonsense Question Answering at Scale
In question answering requiring common sense, language models (e.g., GPT...
research
∙
08/05/2022
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
We present Branch-Train-Merge (BTM), a communication-efficient algorithm...
research
∙
06/10/2022
Measuring the Carbon Intensity of AI in Cloud Instances
By providing unprecedented access to computational resources, cloud comp...
research
∙
05/24/2022
Unsupervised Learning of Hierarchical Conversation Structure
Human conversations can evolve in many different ways, creating challeng...
research
∙
03/16/2022
In-Context Learning for Few-Shot Dialogue State Tracking
Collecting and annotating task-oriented dialogues is time-consuming and ...
research
∙
01/25/2022
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection
Language models increasingly rely on massive web dumps for diverse text ...
research
∙
11/14/2021
Time Waits for No One! Analysis and Challenges of Temporal Misalignment
When an NLP model is trained on text data from one time period and teste...
research
∙
10/06/2021
ABC: Attention with Bounded-memory Control
Transformer architectures have achieved state-of-the-art results on a va...
research
∙
10/01/2021
Expected Validation Performance and Estimation of a Random Variable's Maximum
Research in NLP is often supported by experimental results, and improved...
research
∙
08/31/2021
Sentence Bottleneck Autoencoders from Transformer Language Models
Representation learning for text via pretraining a language model on a l...
research
∙
08/27/2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Since the introduction of the transformer model by Vaswani et al. (2017)...
research
∙
08/11/2021
DEMix Layers: Disentangling Domains for Modular Language Modeling
We introduce a new domain expert mixture (DEMix) layer that enables cond...
research
∙
07/02/2021
Scarecrow: A Framework for Scrutinizing Machine Text
Modern neural text generation systems can produce remarkably fluent and ...
research
∙
06/30/2021
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Human evaluations are typically considered the gold standard in natural ...
research
∙
06/30/2021
On the Power of Saturated Transformers: A View from Circuit Complexity
Transformers have become a standard architecture for many NLP problems. ...
research
∙
06/17/2021
Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study
Biomedical knowledge graphs (KGs) hold rich information on entities such...
research
∙
06/16/2021
Specializing Multilingual Language Models: An Empirical Study
Contextualized word representations from pretrained multilingual languag...
research
∙
05/07/2021
On-the-Fly Controlled Text Generation with Experts and Anti-Experts
Despite recent advances in natural language generation, it remains chall...
research
∙
05/07/2021
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Readers of academic research papers often read with the goal of answerin...
research
∙
04/22/2021
Provable Limitations of Acquiring Meaning from Ungrounded Form: What will Future Language Models Understand?
Language models trained on billions of tokens have recently led to unpre...
research
∙
04/18/2021
Go Forth and Prosper: Language Modeling with Ancient Textual History
We introduce a technique for improving document-level language models (L...
research
∙
03/24/2021
Finetuning Pretrained Transformers into RNNs
Transformers have outperformed recurrent neural networks (RNNs) in natur...
research
∙
03/03/2021
Random Feature Attention
Transformers are state-of-the-art models for a variety of sequence model...
research
∙
01/29/2021
Challenges in Automated Debiasing for Toxic Language Detection
Biased associations have been a challenge in the development of classifi...
research
∙
01/17/2021
GENIE: A Leaderboard for Human-in-the-Loop Evaluation of Text Generation
Leaderboards have eased model development for many NLP datasets by stand...
research
∙
12/31/2020
Shortformer: Better Language Modeling using Shorter Inputs
We explore the benefits of decreasing the input length of transformers. ...
research
∙
12/10/2020
Infusing Finetuning with Semantic Dependencies
For natural language processing systems, two kinds of evidence support t...
research
∙
10/24/2020
Measuring Association Between Labels and Free-Text Rationales
Interpretable NLP has taking increasing interest in ensuring that explan...
research
∙
10/15/2020
Unsupervised Bitext Mining and Translation via Self-trained Contextual Embeddings
We describe an unsupervised method to create pseudo-parallel corpora for...
research
∙
10/15/2020
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs
Natural language rationales could provide intuitive, higher-level explan...
research
∙
10/06/2020