Affective computing using speech and eye gaze: a review and bimodal system proposal for continuous affect prediction

by   Jonny O'Dwyer, et al.

Speech has been a widely used modality in the field of affective computing. Recently however, there has been a growing interest in the use of multi-modal affective computing systems. These multi-modal systems incorporate both verbal and non-verbal features for affective computing tasks. Such multi-modal affective computing systems are advantageous for emotion assessment of individuals in audio-video communication environments such as teleconferencing, healthcare, and education. From a review of the literature, the use of eye gaze features extracted from video is a modality that has remained largely unexploited for continuous affect prediction. This work presents a review of the literature within the emotion classification and continuous affect prediction sub-fields of affective computing for both speech and eye gaze modalities. Additionally, continuous affect prediction experiments using speech and eye gaze modalities are presented. A baseline system is proposed using open source software, the performance of which is assessed on a publicly available audio-visual corpus. Further system performance is assessed in a cross-corpus and cross-lingual experiment. The experimental results suggest that eye gaze is an effective supportive modality for speech when used in a bimodal continuous affect prediction system. The addition of eye gaze to speech in a simple feature fusion framework yields a prediction improvement of 6.13 and 1.62


page 1

page 2

page 3

page 4


Continuous Affect Prediction using Eye Gaze

In recent times, there has been significant interest in the machine reco...

Continuous Affect Prediction Using Eye Gaze and Speech

Affective computing research traditionally focused on labeling a person'...

Eye-based Continuous Affect Prediction

Eye-based information channels include the pupils, gaze, saccades, fixat...

Multi-Modal Continuous Valence And Arousal Prediction in the Wild Using Deep 3D Features and Sequence Modeling

Continuous affect prediction in the wild is a very interesting problem a...

A Multi-Modal Approach to Infer Image Affect

The group affect or emotion in an image of people can be inferred by ext...

Speech, Head, and Eye-based Cues for Continuous Affect Prediction

Continuous affect prediction involves the discrete time-continuous regre...

CalmResponses: Displaying Collective Audience Reactions in Remote Communication

We propose a system displaying audience eye gaze and nod reactions for e...

Please sign up or login with your details

Forgot password? Click here to reset