Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents

by   Uttaran Bhattacharya, et al.

We present Text2Gestures, a transformer-based learning method to interactively generate emotive full-body gestures for virtual agents aligned with natural language text inputs. Our method generates emotionally expressive gestures by utilizing the relevant biomechanical features for body expressions, also known as affective features. We also consider the intended task corresponding to the text and the target virtual agents' intended gender and handedness in our generation pipeline. We train and evaluate our network on the MPI Emotional Body Expressions Database and observe that our network produces state-of-the-art performance in generating gestures for virtual agents aligned with the text for narration or conversation. Our network can generate these gestures at interactive rates on a commodity GPU. We conduct a web-based user study and observe that around 91 gestures to be at least plausible on a five-point Likert Scale. The emotions perceived by the participants from the gestures are also strongly positively correlated with the corresponding intended emotions, with a minimum Pearson coefficient of 0.77 in the valence dimension.


page 1

page 2

page 3

page 4


Storytelling Agents with Personality and Adaptivity

We explore the expression of personality and adaptivity through the gest...

Generating Emotive Gaits for Virtual Agents Using Affect-Based Autoregression

We present a novel autoregression network to generate virtual agents tha...

FVA: Modeling Perceived Friendliness of Virtual Agents Using Movement Characteristics

We present a new approach for improving the friendliness and warmth of a...

Vision based body gesture meta features for Affective Computing

Early detection of psychological distress is key to effective treatment....

A Face-to-Face Neural Conversation Model

Neural networks have recently become good at engaging in dialog. However...

Code Repositories

Please sign up or login with your details

Forgot password? Click here to reset