Multimodal Robot Programming by Demonstration: A Preliminary Exploration

by   Gopika Ajaykumar, et al.

Recent years have seen a growth in the number of industrial robots working closely with end-users such as factory workers. This growing use of collaborative robots has been enabled in part due to the availability of end-user robot programming methods that allow users who are not robot programmers to teach robots task actions. Programming by Demonstration (PbD) is one such end-user programming method that enables users to bypass the complexities of specifying robot motions using programming languages by instead demonstrating the desired robot behavior. Demonstrations are often provided by physically guiding the robot through the motions required for a task action in a process known as kinesthetic teaching. Kinesthetic teaching enables users to directly demonstrate task behaviors in the robot's configuration space, making it a popular end-user robot programming method for collaborative robots known for its low cognitive burden. However, because kinesthetic teaching restricts the programmer's teaching to motion demonstrations, it fails to leverage information from other modalities that humans naturally use when providing physical task demonstrations to one other, such as gaze and speech. Incorporating multimodal information into the traditional kinesthetic programming workflow has the potential to enhance robot learning by highlighting critical aspects of a program, reducing ambiguity, and improving situational awareness for the robot learner and can provide insight into the human programmer's intent and difficulties. In this extended abstract, we describe a preliminary study on multimodal kinesthetic demonstrations and future directions for using multimodal demonstrations to enhance robot learning and user programming experiences.


page 1

page 2

page 3

page 4


A Survey on End-User Robot Programming

As robots interact with a broader range of end-users, end-user robot pro...

Interactive Learning-from-Observation through multimodal human demonstration

Learning-from-Observation (LfO) is a robot teaching framework for progra...

TEAM: a parameter-free algorithm to teach collaborative robots motions from user demonstrations

Collaborative robots (cobots) built to work alongside humans must be abl...

Understanding Acoustic Patterns of Human Teachers Demonstrating Manipulation Tasks to Robots

Humans use audio signals in the form of spoken language or verbal reacti...

Understanding Teacher Gaze Patterns for Robot Learning

Human gaze is known to be a strong indicator of underlying human intenti...

Learning Orbitally Stable Systems for Diagrammatically Teaching

Diagrammatic Teaching is a paradigm for robots to acquire novel skills, ...

Assistive Tele-op: Leveraging Transformers to Collect Robotic Task Demonstrations

Sharing autonomy between robots and human operators could facilitate dat...

Please sign up or login with your details

Forgot password? Click here to reset