ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews

by   Mike D'Arcy, et al.

Revising scientific papers based on peer feedback is a challenging task that requires not only deep scientific knowledge and reasoning, but also the ability to recognize the implicit requests in high-level feedback and to choose the best of many possible ways to update the manuscript in response. We introduce this task for large language models and release ARIES, a dataset of review comments and their corresponding paper edits, to enable training and evaluating models. We study two versions of the task: comment-edit alignment and edit generation, and evaluate several baselines, including GPT-4. We find that models struggle even to identify the edits that correspond to a comment, especially in cases where the comment is phrased in an indirect way or where the edit addresses the spirit of a comment but not the precise request. When tasked with generating edits, GPT-4 often succeeds in addressing comments on a surface level, but it rigidly follows the wording of the feedback rather than the underlying intent, and includes fewer technical details than human-written edits. We hope that our formalization, dataset, and analysis will form a foundation for future work in this area.


page 1

page 2

page 3

page 4


A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Peer reviewing is a central component in the scientific publishing proce...

ALL-IN-ONE: Multi-Task Learning BERT models for Evaluating Peer Assessments

Peer assessment has been widely applied across diverse academic fields o...

Generating Summaries for Scientific Paper Review

The review process is essential to ensure the quality of publications. R...

Exploratory analysis of text duplication in peer-review reveals peer-review fraud and paper mills

Comments received from referees during peer-review were analysed to dete...

Sentence-level Feedback Generation for English Language Learners: Does Data Augmentation Help?

In this paper, we present strong baselines for the task of Feedback Comm...

Argument Mining for Understanding Peer Reviews

Peer-review plays a critical role in the scientific writing and publicat...

Peer Reviewing Revisited: Assessing Research with Interlinked Semantic Comments

Scientific publishing seems to be at a turning point. Its paradigm has s...

Please sign up or login with your details

Forgot password? Click here to reset