CogniVal: A Framework for Cognitive Word Embedding Evaluation

by   Nora Hollenstein, et al.
Universität Zürich
ETH Zurich

An interesting method of evaluating word representations is by how much they reflect the semantic representations in the human brain. However, most, if not all, previous works only focus on small datasets and a single modality. In this paper, we present the first multi-modal framework for evaluating English word representations based on cognitive lexical semantics. Six types of word embeddings are evaluated by fitting them to 15 datasets of eye-tracking, EEG and fMRI signals recorded during language processing. To achieve a global score over all evaluation hypotheses, we apply statistical significance testing accounting for the multiple comparisons problem. This framework is easily extensible and available to include other intrinsic and extrinsic evaluation methods. We find strong correlations in the results between cognitive datasets, across recording modalities and to their performance on extrinsic NLP tasks.


page 1

page 2

page 3

page 4


CogniFNN: A Fuzzy Neural Network Framework for Cognitive Word Embedding Evaluation

Word embeddings can reflect the semantic representations, and the embedd...

CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals

Most previous studies integrate cognitive language processing signals (e...

Advancing NLP with Cognitive Language Processing Signals

When we read, our brain processes language and generates cognitive proce...

Bridging between Cognitive Processing Signals and Linguistic Features via a Unified Attentional Network

Cognitive processing signals can be used to improve natural language pro...

A Survey of Word Embeddings Evaluation Methods

Word embeddings are real-valued word representations able to capture lex...

How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

Maybe the single most important goal of representation learning is makin...

Intrinsic Subspace Evaluation of Word Embedding Representations

We introduce a new methodology for intrinsic evaluation of word represen...

Please sign up or login with your details

Forgot password? Click here to reset