StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models

by   Adam Liska, et al.

Knowledge and language understanding of models evaluated through question answering (QA) has been usually studied on static snapshots of knowledge, like Wikipedia. However, our world is dynamic, evolves over time, and our models' knowledge becomes outdated. To study how semi-parametric QA models and their underlying parametric language models (LMs) adapt to evolving knowledge, we construct a new large-scale dataset, StreamingQA, with human written and generated questions asked on a given date, to be answered from 14 years of time-stamped news articles. We evaluate our models quarterly as they read new articles not seen in pre-training. We show that parametric models can be updated without full retraining, while avoiding catastrophic forgetting. For semi-parametric models, adding new articles into the search space allows for rapid adaptation, however, models with an outdated underlying LM under-perform those with a retrained LM. For questions about higher-frequency named entities, parametric updates are particularly beneficial. In our dynamic world, the StreamingQA dataset enables a more realistic evaluation of QA models, and our experiments highlight several promising directions for future research.


DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering

Question answering models commonly have access to two sources of "knowle...

ForecastQA: Machine Comprehension of Temporal Text for Answering Forecasting Questions

Textual data are often accompanied by time information (e.g., dates in n...

Plug-and-Play Adaptation for Continuously-updated QA

Language models (LMs) have shown great potential as implicit knowledge b...

Entity-Based Knowledge Conflicts in Question Answering

Knowledge-dependent tasks typically use two sources of knowledge: parame...

ScanQA: 3D Question Answering for Spatial Scene Understanding

We propose a new 3D spatial understanding task of 3D Question Answering ...

Entity Cloze By Date: What LMs Know About Unseen Entities

Language models (LMs) are typically trained once on a large-scale corpus...

Question Answering Over Biological Knowledge Graph via Amazon Alexa

Structured and unstructured data and facts about drugs, genes, protein, ...

Please sign up or login with your details

Forgot password? Click here to reset