Semantic Sensing and Planning for Human-Robot Collaboration in Uncertain Environments

by   Luke Burks, et al.

Autonomous robots can benefit greatly from human-provided semantic characterizations of uncertain task environments and states. However, the development of integrated strategies which let robots model, communicate, and act on such soft data remains challenging. Here, a framework is presented for active semantic sensing and planning in human-robot teams which addresses these gaps by formally combining the benefits of online sampling-based POMDP policies, multi-modal semantic interaction, and Bayesian data fusion. This approach lets humans opportunistically impose model structure and extend the range of semantic soft data in uncertain environments by sketching and labeling arbitrary landmarks across the environment. Dynamic updating of the environment while searching for a mobile target allows robotic agents to actively query humans for novel and relevant semantic data, thereby improving beliefs of unknown environments and target states for improved online planning. Target search simulations show significant improvements in time and belief state estimates required for interception versus conventional planning based solely on robotic sensing. Human subject studies demonstrate a average doubling in dynamic target capture rate compared to the lone robot case, employing reasoning over a range of user characteristics and interaction modalities. Video of interaction can be found at


page 2

page 6

page 9

page 10

page 11

page 16

page 21

page 22


Probabilistic Semantic Data Association for Collaborative Human-Robot Sensing

In collaborative human-robot semantic sensing problems, e.g. for scienti...

Multi-robot Mission Planning in Dynamic Semantic Environments

This paper addresses a new semantic multi-robot planning problem in unce...

Semantic-Aware Environment Perception for Mobile Human-Robot Interaction

Current technological advances open up new opportunities for bringing hu...

Improving Tracking through Human-Robot Sensory Augmentation

This paper introduces human-robot sensory augmentation and illustrates i...

Closed-loop Bayesian Semantic Data Fusion for Collaborative Human-Autonomy Target Search

In search applications, autonomous unmanned vehicles must be able to eff...

Distributed Hierarchical Control for State Estimation With Robotic Sensor Networks

This paper addresses active state estimation with a team of robotic sens...

Robot Active Neural Sensing and Planning in Unknown Cluttered Environments

Active sensing and planning in unknown, cluttered environments is an ope...

Please sign up or login with your details

Forgot password? Click here to reset