Long Activity Video Understanding using Functional Object-Oriented Network

by   Ahmad Babaeian Jelodar, et al.

Video understanding is one of the most challenging topics in computer vision. In this paper, a four-stage video understanding pipeline is presented to simultaneously recognize all atomic actions and the single on-going activity in a video. This pipeline uses objects and motions from the video and a graph-based knowledge representation network as prior reference. Two deep networks are trained to identify objects and motions in each video sequence associated with an action. Low Level image features are then used to identify objects of interest in that video sequence. Confidence scores are assigned to objects of interest based on their involvement in the action and to motion classes based on results from a deep neural network that classifies the on-going action in video into motion classes. Confidence scores are computed for each candidate functional unit associated with an action using a knowledge representation network, object confidences, and motion confidences. Each action is therefore associated with a functional unit and the sequence of actions is further evaluated to identify the single on-going activity in the video. The knowledge representation used in the pipeline is called the functional object-oriented network which is a graph-based network useful for encoding knowledge about manipulation tasks. Experiments are performed on a dataset of cooking videos to test the proposed algorithm with action inference and activity classification. Experiments show that using functional object oriented network improves video understanding significantly.


page 6

page 9

page 10


Functional Object-Oriented Network for Manipulation Learning

This paper presents a novel structured knowledge representation called t...

Functional Object-Oriented Network: Construction & Expansion

We build upon the functional object-oriented network (FOON), a structure...

Long-Horizon Task and Motion Planning with Functional Object-Oriented Networks

Following work on joint object-action representations, functional object...

A Structured Model For Action Detection

A dominant paradigm for learning-based approaches in computer vision is ...

DAP3D-Net: Where, What and How Actions Occur in Videos?

Action parsing in videos with complex scenes is an interesting but chall...

Prediction of Manipulation Actions

Looking at a person's hands one often can tell what the person is going ...

Grounding of the Functional Object-Oriented Network in Industrial Tasks

In this preliminary work, we propose to design an activity recognition s...

Please sign up or login with your details

Forgot password? Click here to reset