Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering – Example of ChatGPT

by   Maciej P. Polak, et al.

There has been a growing effort to replace hand extraction of data from research papers with automated data extraction based on natural language processing (NLP), language models (LMs), and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with essentially no initial effort or background using an advanced conversational LLM (or AI). ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract data, and assure its correctness through a series of follow-up questions. These follow-up questions address a critical challenge associated with LLMs - their tendency to provide factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both over 90 likely rivaling or exceeding human accuracy in many cases. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability and accuracy are likely to replace other methods of data extraction in the near future.


Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models

Accurate and comprehensive material databases extracted from research pa...

Chit-Chat or Deep Talk: Prompt Engineering for Process Mining

This research investigates the application of Large Language Models (LLM...

Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT

The amount of data has growing significance in exploring cutting-edge ma...

Towards Autonomous Testing Agents via Conversational Large Language Models

Software testing is an important part of the development cycle, yet it r...

What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys

Conversational surveys, where an agent asks open-ended questions through...

KnowGraph@IITK at SemEval-2021 Task 11: Building KnowledgeGraph for NLP Research

Research in Natural Language Processing is making rapid advances, result...

SkillGPT: a RESTful API service for skill extraction and standardization using a Large Language Model

We present SkillGPT, a tool for skill extraction and standardization (SE...

Please sign up or login with your details

Forgot password? Click here to reset