FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering

by   Akhil Kedia, et al.

Generative models have recently started to outperform extractive models in Open Domain Question Answering, largely by leveraging their decoder to attend over multiple encoded passages and combining their information. However, generative models tend to be larger than extractive models due to the need for a decoder, run slower during inference due to auto-regressive decoder beam search, and their generated output often suffers from hallucinations. We propose to extend transformer encoders with the ability to fuse information from multiple passages, using global representation to provide cross-sample attention over all tokens across samples. Furthermore, we propose an alternative answer span probability calculation to better aggregate answer scores in the global space of all samples. Using our proposed method, we outperform the current state-of-the-art method by 2.5 Exact Match score on the Natural Question dataset while using only 25% of parameters and 35% of the latency during inference, and 4.4 Exact Match on WebQuestions dataset. When coupled with synthetic data augmentation, we outperform larger models on the TriviaQA dataset as well. The latency and parameter savings of our method make it particularly attractive for open-domain question answering, as these models are often compute-intensive.


page 1

page 2

page 3

page 4


Attention-guided Generative Models for Extractive Question Answering

We propose a novel method for applying Transformer models to extractive ...

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

Generative models for open domain question answering have proven to be c...

RFiD: Towards Rational Fusion-in-Decoder for Open-Domain Question Answering

Open-Domain Question Answering (ODQA) systems necessitate a reader model...

You Only Need One Model for Open-domain Question Answering

Recent works for Open-domain Question Answering refer to an external kno...

KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering

Current Open-Domain Question Answering (ODQA) model paradigm often conta...

SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs

In this work, we present SciGraphQA, a synthetic multi-turn question-ans...

Defending Against Poisoning Attacks in Open-Domain Question Answering

Recent work in open-domain question answering (ODQA) has shown that adve...

Please sign up or login with your details

Forgot password? Click here to reset