A Closer Look at Temporal Ordering in the Segmentation of Instructional Videos

by   Anil Batra, et al.

Understanding the steps required to perform a task is an important skill for AI systems. Learning these steps from instructional videos involves two subproblems: (i) identifying the temporal boundary of sequentially occurring segments and (ii) summarizing these steps in natural language. We refer to this task as Procedure Segmentation and Summarization (PSS). In this paper, we take a closer look at PSS and propose three fundamental improvements over current methods. The segmentation task is critical, as generating a correct summary requires the step to be identified first. However, current segmentation metrics often overestimate the segmentation quality because they do not incorporate the temporal order of segments. We propose a new segmentation metric based on dynamic programming that takes into account the order of segments. Current PSS methods are typically trained by proposing segments, matching them with the ground truth and computing a loss. However, much like segmentation metrics, existing matching algorithms do not consider the temporal order of the mapping between candidate segments and the ground truth. We propose a matching algorithm that constrains the temporal order of segment mapping, and is also differentiable. Lastly, we introduce multi-modal feature training for PSS, which further improves segmentation. We evaluate our approach on two instructional video datasets (YouCook2 and Tasty) and improve the state of the art by a margin of ∼7% and ∼2.5% for procedure segmentation and summarization, respectively.


TL;DW? Summarizing Instructional Videos with Task Relevance Cross-Modal Saliency

YouTube users looking for instructions for a specific task may spend a l...

Unsupervised Human Action Detection by Action Matching

We propose a new task of unsupervised action detection by action matchin...

Realistic Video Summarization through VISIOCITY: A New Benchmark and Evaluation Framework

Automatic video summarization is still an unsolved problem due to severa...

Visual Summarization of Lecture Video Segments for Enhanced Navigation

Lecture videos are an increasingly important learning resource for highe...

Supervized Segmentation with Graph-Structured Deep Metric Learning

We present a fully-supervized method for learning to segment data struct...

Towards Automatic Learning of Procedures from Web Instructional Videos

The potential for agents, whether embodied or software, to learn by obse...

A Closer Look at Temporal Sentence Grounding in Videos: Datasets and Metrics

Despite Temporal Sentence Grounding in Videos (TSGV) has realized impres...

Please sign up or login with your details

Forgot password? Click here to reset