RERERE: Remote Embodied Referring Expressions in Real indoor Environments

by   Yuankai Qi, et al.

One of the long-term challenges of robotics is to enable humans to communicate with robots about the world. It is essential if they are to collaborate. Humans are visual animals, and we communicate primarily through language, so human-robot communication is inevitably at least partly a vision-and-language problem. This has motivated both Referring Expression datasets, and Vision and Language Navigation datasets. These partition the problem into that of identifying an object of interest, or navigating to another location. Many of the most appealing uses of robots, however, require communication about remote objects and thus do not reflect the dichotomy in the datasets. We thus propose the first Remote Embodied Referring Expression dataset of natural language references to remote objects in real images. Success requires navigating through a previously unseen environment to select an object identified through general natural language. This represents a complex challenge, but one that closely reflects one of the core visual problems in robotics. A Navigator-Pointer model which provides a strong baseline on the task is also proposed.


page 1

page 3

page 6

page 12

page 13

page 14

page 15

page 16


Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction

The human language is one of the most natural interfaces for humans to i...

RREx-BoT: Remote Referring Expressions with a Bag of Tricks

Household robots operate in the same space for years. Such robots increm...

CLIPUNetr: Assisting Human-robot Interface for Uncalibrated Visual Servoing Control with CLIP-driven Referring Expression Segmentation

The classical human-robot interface in uncalibrated image-based visual s...

Accessible Interfaces for the Development and Deployment of Robotic Platforms

Accessibility is one of the most important features in the design of rob...

Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions

A long-term goal of AI research is to build intelligent agents that can ...

Natural Language for Human-Robot Collaboration: Problems Beyond Language Grounding

To enable robots to instruct humans in collaborations, we identify sever...

Natural Language Interaction to Facilitate Mental Models of Remote Robots

Increasingly complex and autonomous robots are being deployed in real-wo...

Please sign up or login with your details

Forgot password? Click here to reset