Affection: Learning Affective Explanations for Real-World Visual Data

by   Panos Achlioptas, et al.

In this work, we explore the emotional reactions that real-world images tend to induce by using natural language as the medium to express the rationale behind an affective response to a given visual stimulus. To embark on this journey, we introduce and share with the research community a large-scale dataset that contains emotional reactions and free-form textual explanations for 85,007 publicly available images, analyzed by 6,283 annotators who were asked to indicate and explain how and why they felt in a particular way when observing a specific image, producing a total of 526,749 responses. Even though emotional reactions are subjective and sensitive to context (personal mood, social status, past experiences) - we show that there is significant common ground to capture potentially plausible emotional responses with a large support in the subject population. In light of this crucial observation, we ask the following questions: i) Can we develop multi-modal neural networks that provide reasonable affective responses to real-world visual data, explained with language? ii) Can we steer such methods towards producing explanations with varying degrees of pragmatic language or justifying different emotional reactions while adapting to the underlying visual stimulus? Finally, iii) How can we evaluate the performance of such methods for this novel task? With this work, we take the first steps in addressing all of these questions, thus paving the way for richer, more human-centric, and emotionally-aware image analysis systems. Our introduced dataset and all developed methods are available on


page 4

page 7

page 9

page 10

page 11

page 13


ArtEmis: Affective Language for Visual Art

We present a novel large-scale dataset and accompanying machine learning...

CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations

Providing explanations in the context of Visual Question Answering (VQA)...

Measuring Emotions in the COVID-19 Real World Worry Dataset

The COVID-19 pandemic is having a dramatic impact on societies and econo...

Learning to Explain: Answering Why-Questions via Rephrasing

Providing plausible responses to why questions is a challenging but crit...

e-SNLI-VE-2.0: Corrected Visual-Textual Entailment with Natural Language Explanations

The recently proposed SNLI-VE corpus for recognising visual-textual enta...

Building Emotional Support Chatbots in the Era of LLMs

The integration of emotional support into various conversational scenari...

Determination of the Internet Anonymity Influence on the Level of Aggression and Usage of Obscene Lexis

This article deals with the analysis of the semantic content of the anon...

Please sign up or login with your details

Forgot password? Click here to reset