Knowledge Graph Prompting for Multi-Document Question Answering

by   Yu Wang, et al.

The 'pre-train, prompt, predict' paradigm of large language models (LLMs) has achieved remarkable success in open-domain question answering (OD-QA). However, few works explore this paradigm in the scenario of multi-document question answering (MD-QA), a task demanding a thorough understanding of the logical associations among the contents and structures of different documents. To fill this crucial gap, we propose a Knowledge Graph Prompting (KGP) method to formulate the right context in prompting LLMs for MD-QA, which consists of a graph construction module and a graph traversal module. For graph construction, we create a knowledge graph (KG) over multiple documents with nodes symbolizing passages or document structures (e.g., pages/tables), and edges denoting the semantic/lexical similarity between passages or intra-document structural relations. For graph traversal, we design an LM-guided graph traverser that navigates across nodes and gathers supporting passages assisting LLMs in MD-QA. The constructed graph serves as the global ruler that regulates the transitional space among passages and reduces retrieval latency. Concurrently, the LM-guided traverser acts as a local navigator that gathers pertinent context to progressively approach the question and guarantee retrieval quality. Extensive experiments underscore the efficacy of KGP for MD-QA, signifying the potential of leveraging graphs in enhancing the prompt design for LLMs. Our code is at


page 18

page 19

page 20

page 21

page 22


DDRQA: Dynamic Document Reranking for Open-domain Multi-hop Question Answering

Open-domain multi-hop question answering (QA) requires to retrieve multi...

Knowledge-Aided Open-Domain Question Answering

Open-domain question answering (QA) aims to find the answer to a questio...

PDFTriage: Question Answering over Long, Structured Documents

Large Language Models (LLMs) have issues with document question answerin...

Knowledge Graph-based Question Answering with Electronic Health Records

Question Answering (QA) on Electronic Health Records (EHR), namely EHR Q...

UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph

Multi-hop Question Answering over Knowledge Graph (KGQA) aims to find th...

Textbook Question Answering with Knowledge Graph Understanding and Unsupervised Open-set Text Comprehension

In this work, we introduce a novel algorithm for solving the textbook qu...

Capturing Global Structural Information in Long Document Question Answering with Compressive Graph Selector Network

Long document question answering is a challenging task due to its demand...

Please sign up or login with your details

Forgot password? Click here to reset