REPLUG: Retrieval-Augmented Black-Box Language Models

by   Weijia Shi, et al.

We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing retrieval and language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improves the performance of GPT-3 (175B) on language modeling by 6.3 5.1


kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Models

Fine-tuning a language model on a new domain is standard practice for do...

In-Context Retrieval-Augmented Language Models

Retrieval-Augmented Language Modeling (RALM) methods, that condition a l...

Long-range Language Modeling with Self-retrieval

Retrieval-augmented language models (LMs) have received much attention r...

Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

Retrieval-based language models (R-LM) model the probability of natural ...

PROPS: Probabilistic personalization of black-box sequence models

We present PROPS, a lightweight transfer learning mechanism for sequenti...

Watermarking Text Generated by Black-Box Language Models

LLMs now exhibit human-like skills in various fields, leading to worries...

Hybrid Retrieval-Augmented Generation for Real-time Composition Assistance

Retrieval augmented models show promise in enhancing traditional languag...

Please sign up or login with your details

Forgot password? Click here to reset