Human brain activity for machine attention

by   Lukas Muttenthaler, et al.
Københavns Uni
ETH Zurich

Cognitively inspired NLP leverages human-derived data to teach machines about language processing mechanisms. Recently, neural networks have been augmented with behavioral data to solve a range of NLP tasks spanning syntax and semantics. We are the first to exploit neuroscientific data, namely electroencephalography (EEG), to inform a neural attention model about language processing of the human brain with direct cognitive measures. Part of the challenge in working with EEG is that features are exceptionally rich and need extensive pre-processing to isolate signals specific to text processing. We devise a method for finding such EEG features to supervise machine attention through combining theoretically motivated cropping with random forest tree splits. This method considerably reduces the number of dimensions of the EEG data. We employ the method on a publicly available EEG corpus and demonstrate that the pre-processed EEG features are capable of distinguishing two reading tasks. We apply these features to regularise attention on relation classification and show that EEG is more informative than strong baselines. This improvement, however, is dependent on both the cognitive load of the task and the EEG frequency domain. Hence, informing neural attention models through EEG signals has benefits but requires further investigation to understand which dimensions are most useful across NLP tasks.


Advancing NLP with Cognitive Language Processing Signals

When we read, our brain processes language and generates cognitive proce...

Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

Until recently, human behavioral data from reading has mainly been of in...

CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals

Most previous studies integrate cognitive language processing signals (e...

Synthesizing Human Gaze Feedback for Improved NLP Performance

Integrating human feedback in models can improve the performance of natu...

An Empirical Exploration of Cross-domain Alignment between Language and Electroencephalogram

Electroencephalography (EEG) and language have been widely explored inde...

Bayesian Topological Learning for Brain State Classification

Investigation of human brain states through electroencephalograph (EEG) ...

Please sign up or login with your details

Forgot password? Click here to reset