Learning Human Search Behavior from Egocentric Visual Inputs

by   Maks Sorokin, et al.

"Looking for things" is a mundane but critical task we repeatedly carry on in our daily life. We introduce a method to develop a human character capable of searching for a randomly located target object in a detailed 3D scene using its locomotion capability and egocentric vision perception represented as RGBD images. By depriving the privileged 3D information from the human character, it is forced to move and look around simultaneously to account for the restricted sensing capability, resulting in natural navigation and search behaviors. Our method consists of two components: 1) a search control policy based on an abstract character model, and 2) an online replanning control module for synthesizing detailed kinematic motion based on the trajectories planned by the search policy. We demonstrate that the combined techniques enable the character to effectively find often occluded household items in indoor environments. The same search policy can be applied to different full-body characters without the need for retraining. We evaluate our method quantitatively by testing it on randomly generated scenarios. Our work is a first step toward creating intelligent virtual agents with humanlike behaviors driven by onboard sensors, paving the road toward future robotic applications.


page 1

page 6

page 8


GAPLE: Generalizable Approaching Policy LEarning for Robotic Object Searching in Indoor Environment

We study the problem of learning a generalizable action policy for an in...

Synthesizing Physical Character-Scene Interactions

Movement is how people interact with and affect their environment. For r...

A Survey on Reinforcement Learning Methods in Character Animation

Reinforcement Learning is an area of Machine Learning focused on how age...

Follow Alice into the Rabbit Hole: Giving Dialogue Agents Understanding of Human Level Attributes

For conversational AI and virtual assistants to communicate with humans ...

Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments

Synthesizing interaction-involved human motions has been challenging due...

Realistic Physics Based Character Controller

Over the course of the last several years there was a strong interest in...

Full-Body Locomotion Reconstruction of Virtual Characters Using a Single IMU

This paper presents a method of reconstructing full-body locomotion sequ...

Please sign up or login with your details

Forgot password? Click here to reset