What Should the System Do Next?: Operative Action Captioning for Estimating System Actions

by   Taiki Nakamura, et al.

Such human-assisting systems as robots need to correctly understand the surrounding situation based on observations and output the required support actions for humans. Language is one of the important channels to communicate with humans, and the robots are required to have the ability to express their understanding and action planning results. In this study, we propose a new task of operative action captioning that estimates and verbalizes the actions to be taken by the system in a human-assisting domain. We constructed a system that outputs a verbal description of a possible operative action that changes the current state to the given target state. We collected a dataset consisting of two images as observations, which express the current state and the state changed by actions, and a caption that describes the actions that change the current state to the target state, by crowdsourcing in daily life situations. Then we constructed a system that estimates operative action by a caption. Since the operative action's caption is expected to contain some state-changing actions, we use scene-graph prediction as an auxiliary task because the events written in the scene graphs correspond to the state changes. Experimental results showed that our system successfully described the operative actions that should be conducted between the current and target states. The auxiliary tasks that predict the scene graphs improved the quality of the estimation results.


page 1

page 2

page 4

page 6


Abductive Action Inference

Abductive reasoning aims to make the most likely inference for a given s...

Action Prediction in Humans and Robots

Efficient action prediction is of central importance for the fluent work...

Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions

In-depth scene descriptions and question answering tasks have greatly in...

Action Quality Assessment Across Multiple Actions

Can learning to measure the quality of an action help in measuring the q...

A Learning-Based Method for Automatic Operator Selection in the Fanoos XAI System

We describe an extension of the Fanoos XAI system [Bayani et al 2022] wh...

Predictor models for high-performance wheel loading

Autonomous wheel loading involves selecting actions that maximize the to...

Explainable Object-induced Action Decision for Autonomous Vehicles

A new paradigm is proposed for autonomous driving. The new paradigm lies...

Please sign up or login with your details

Forgot password? Click here to reset