DeepAI AI Chat
Log In Sign Up

Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

by   Nora Hollenstein, et al.

Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity to this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, further research is needed. Finally, EEG data shows to be particularly promising when limited training data is available.


page 1

page 2

page 3

page 4


Human brain activity for machine attention

Cognitively inspired NLP leverages human-derived data to teach machines ...

Decoding speech from non-invasive brain recordings

Decoding language from brain activity is a long-awaited goal in both hea...

Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

State-of-the-art brain-to-text systems have achieved great success in de...

Features and Machine Learning for Correlating and Classifying between Brain Areas and Dyslexia

We develop a method that is based on processing gathered Event Related P...

Prediction of Human Empathy based on EEG Cortical Asymmetry

Humans constantly interact with digital devices that disregard their fee...

An Empirical Exploration of Cross-domain Alignment between Language and Electroencephalogram

Electroencephalography (EEG) and language have been widely explored inde...

A smile is all you need: Predicting limiting activity coefficients from SMILES with natural language processing

Knowledge of mixtures' phase equilibria is crucial in nature and technic...