How Does Generative Retrieval Scale to Millions of Passages?

by   Ronak Pradeep, et al.

Popularized by the Differentiable Search Index, the emerging paradigm of generative retrieval re-frames the classic information retrieval problem into a sequence-to-sequence modeling task, forgoing external indices and encoding an entire document corpus within a single Transformer. Although many different approaches have been proposed to improve the effectiveness of generative retrieval, they have only been evaluated on document corpora on the order of 100k in size. We conduct the first empirical study of generative retrieval techniques across various corpus scales, ultimately scaling up to the entire MS MARCO passage ranking task with a corpus of 8.8M passages and evaluating model sizes up to 11B parameters. We uncover several findings about scaling generative retrieval to millions of passages; notably, the central importance of using synthetic queries as document representations during indexing, the ineffectiveness of existing proposed architecture modifications when accounting for compute cost, and the limits of naively scaling model parameters with respect to retrieval performance. While we find that generative retrieval is competitive with state-of-the-art dual encoders on small corpora, scaling to millions of passages remains an important and unsolved challenge. We believe these findings will be valuable for the community to clarify the current state of generative retrieval, highlight the unique challenges, and inspire new research directions.


page 1

page 2

page 3

page 4


Generative Retrieval as Dense Retrieval

Generative retrieval is a promising new neural retrieval paradigm that a...

Learning to Tokenize for Generative Retrieval

Conventional document retrieval techniques are mainly based on the index...

A Neural Corpus Indexer for Document Retrieval

Current state-of-the-art document retrieval solutions mainly follow an i...

DSI++: Updating Transformer Memory with New Documents

Differentiable Search Indices (DSIs) encode a corpus of documents in the...

Doc2Query–: When Less is More

Doc2Query – the process of expanding the content of a document before in...

Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies

Recently, a new paradigm called Differentiable Search Index (DSI) has be...

Continually Updating Generative Retrieval on Dynamic Corpora

Generative retrieval has recently been gaining a lot of attention from t...

Please sign up or login with your details

Forgot password? Click here to reset