Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study

by   Farahnaz Akrami, et al.

In the active research area of employing embedding models for knowledge graph completion, particularly for the task of link prediction, most prior studies used two benchmark datasets FB15k and WN18 in evaluating such models. Most triples in these and other datasets in such studies belong to reverse and duplicate relations which exhibit high data redundancy due to semantic duplication, correlation or data incompleteness. This is a case of excessive data leakage—a model is trained using features that otherwise would not be available when the model needs to be applied for real prediction. There are also Cartesian product relations for which every triple formed by the Cartesian product of applicable subjects and objects is a true fact. Link prediction on the aforementioned relations is easy and can be achieved with even better accuracy using straightforward rules instead of sophisticated embedding models. A more fundamental defect of these models is that the link prediction scenario, given such data, is non-existent in the real-world. This paper is the first systematic study with the main objective of assessing the true effectiveness of embedding models when the unrealistic triples are removed. Our experiment results show these models are much less accurate than what we used to perceive. Their poor accuracy renders link prediction a task without truly effective automated solution. Hence, we call for re-investigation of possible effective approaches.


Farspredict: A benchmark dataset for link prediction

Link prediction with knowledge graph embedding (KGE) is a popular method...

KGRefiner: Knowledge Graph Refinement for Improving Accuracy of Translational Link Prediction Methods

Link prediction is the task of predicting missing relations between enti...

Orthogonal Relation Transforms with Graph Context Modeling for Knowledge Graph Embedding

Translational distance-based knowledge graph embedding has shown progres...

Knowledge Graph Embedding Bi-Vector Models for Symmetric Relation

Knowledge graph embedding (KGE) models have been proposed to improve the...

Predicting Semantic Relations using Global Graph Properties

Semantic graphs, such as WordNet, are resources which curate natural lan...

Runtime Performances Benchmark for Knowledge Graph Embedding Methods

This paper wants to focus on providing a characterization of the runtime...

A Systematic Investigation of KB-Text Embedding Alignment at Scale

Knowledge bases (KBs) and text often contain complementary knowledge: KB...

Please sign up or login with your details

Forgot password? Click here to reset