The EMPATHIC Framework for Task Learning from Implicit Human Feedback

by   Yuchen Cui, et al.

Reactions such as gestures, facial expressions, and vocalizations are an abundant, naturally occurring channel of information that humans provide during interactions. A robot or other agent could leverage an understanding of such implicit human feedback to improve its task performance at no cost to the human. This approach contrasts with common agent teaching methods based on demonstrations, critiques, or other guidance that need to be attentively and intentionally provided. In this paper, we first define the general problem of learning from implicit human feedback and then propose to address this problem through a novel data-driven framework, EMPATHIC. This two-stage method consists of (1) mapping implicit human feedback to relevant task statistics such as reward, optimality, and advantage; and (2) using such a mapping to learn a task. We instantiate the first stage and three second-stage evaluations of the learned mapping. To do so, we collect a dataset of human facial reactions while participants observe an agent execute a sub-optimal policy for a prescribed training task. We train a deep neural network on this data and demonstrate its ability to (1) infer relative reward ranking of events in the training task from prerecorded human facial reactions; (2) improve the policy of an agent in the training task using live human facial reactions; and (3) transfer to a novel domain in which it evaluates robot manipulation trajectories.


page 4

page 5

page 6

page 13

page 14

page 16

page 17


Convergence of a Human-in-the-Loop Policy-Gradient Algorithm With Eligibility Trace Under Reward, Policy, and Advantage Feedback

Fluid human-agent communication is essential for the future of human-in-...

Facial Feedback for Reinforcement Learning: A Case Study and Offline Analysis Using the TAMER Framework

Interactive reinforcement learning provides a way for agents to learn to...

Face valuing: Training user interfaces with facial expressions and reinforcement learning

An important application of interactive machine learning is extending or...

Learning from Observations Using a Single Video Demonstration and Human Feedback

In this paper, we present a method for learning from video demonstration...

Learning via social awareness: improving sketch representations with facial feedback

In the quest towards general artificial intelligence (AI), researchers h...

Feedback-efficient Active Preference Learning for Socially Aware Robot Navigation

Socially aware robot navigation, where a robot is required to optimize i...

Learning Behaviors with Uncertain Human Feedback

Human feedback is widely used to train agents in many domains. However, ...

Please sign up or login with your details

Forgot password? Click here to reset