Explainable Video Action Reasoning via Prior Knowledge and State Transitions

by   Tao Zhuo, et al.

Human action analysis and understanding in videos is an important and challenging task. Although substantial progress has been made in past years, the explainability of existing methods is still limited. In this work, we propose a novel action reasoning framework that uses prior knowledge to explain semantic-level observations of video state changes. Our method takes advantage of both classical reasoning and modern deep learning approaches. Specifically, prior knowledge is defined as the information of a target video domain, including a set of objects, attributes and relationships in the target video domain, as well as relevant actions defined by the temporal attribute and relationship changes (i.e. state transitions). Given a video sequence, we first generate a scene graph on each frame to represent concerned objects, attributes and relationships. Then those scene graphs are linked by tracking objects across frames to form a spatio-temporal graph (also called video graph), which represents semantic-level video states. Finally, by sequentially examining each state transition in the video graph, our method can detect and explain how those actions are executed with prior knowledge, just like the logical manner of thinking by humans. Compared to previous works, the action reasoning results of our method can be explained by both logical rules and semantic-level observations of video content changes. Besides, the proposed method can be used to detect multiple concurrent actions with detailed information, such as who (particular objects), when (time), where (object locations) and how (what kind of changes). Experiments on a re-annotated dataset CAD-120 show the effectiveness of our method.


page 1

page 4

page 6

page 7


Unified Graph Structured Models for Video Understanding

Accurate video understanding involves reasoning about the relationships ...

Spatio-Temporal Interaction Graph Parsing Networks for Human-Object Interaction Recognition

For a given video-based Human-Object Interaction scene, modeling the spa...

Relational Reasoning using Prior Knowledge for Visual Captioning

Exploiting relationships among objects has achieved remarkable progress ...

Classification by Attention: Scene Graph Classification with Prior Knowledge

A main challenge in scene graph classification is that the appearance of...

Reasoning over the Behaviour of Objects in Video-Clips for Adverb-Type Recognition

In this work, following the intuition that adverbs describing scene-sequ...

Abductive Action Inference

Abductive reasoning aims to make the most likely inference for a given s...

Explainable Object-induced Action Decision for Autonomous Vehicles

A new paradigm is proposed for autonomous driving. The new paradigm lies...

Please sign up or login with your details

Forgot password? Click here to reset