PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation

by   Ishan Jindal, et al.

Semantic role labeling (SRL) identifies the predicate-argument structure in a sentence. This task is usually accomplished in four steps: predicate identification, predicate sense disambiguation, argument identification, and argument classification. Errors introduced at one step propagate to later steps. Unfortunately, the existing SRL evaluation scripts do not consider the full effect of this error propagation aspect. They either evaluate arguments independent of predicate sense (CoNLL09) or do not evaluate predicate sense at all (CoNLL05), yielding an inaccurate SRL model performance on the argument classification task. In this paper, we address key practical issues with existing evaluation scripts and propose a more strict SRL evaluation metric PriMeSRL. We observe that by employing PriMeSRL, the quality evaluation of all SoTA SRL models drops significantly, and their relative rankings also change. We also show that PriMeSRLsuccessfully penalizes actual failures in SoTA SRL models.


page 1

page 2

page 3

page 4


Neural Unsupervised Semantic Role Labeling

The task of semantic role labeling (SRL) is dedicated to finding the pre...

Argument Linking: A Survey and Forecast

Semantic role labeling (SRL) – identifying the semantic relationships be...

An MRC Framework for Semantic Role Labeling

Semantic Role Labeling (SRL) aims at recognizing the predicate-argument ...

I Learn to Diffuse, or Data Alchemy 101: a Mnemonic Manifesto

In this manifesto, we put forward the idea of data alchemy as a narrativ...

CLAR: A Cross-Lingual Argument Regularizer for Semantic Role Labeling

Semantic role labeling (SRL) identifies predicate-argument structure(s) ...

Persian Semantic Role Labeling Using Transfer Learning and BERT-Based Models

Semantic role labeling (SRL) is the process of detecting the predicate-a...

Please sign up or login with your details

Forgot password? Click here to reset