How Good is a Video Summary? A New Benchmarking Dataset and Evaluation Framework Towards Realistic Video Summarization

by   Vishal Kaushal, et al.

Automatic video summarization is still an unsolved problem due to several challenges. The currently available datasets either have very short videos or have few long videos of only a particular type. We introduce a new benchmarking video dataset called VISIOCITY (VIdeo SummarIzatiOn based on Continuity, Intent and DiversiTY) which comprises of longer videos across six different categories with dense concept annotations capable of supporting different flavors of video summarization and other vision problems. For long videos, human reference summaries necessary for supervised video summarization techniques are difficult to obtain. We explore strategies to automatically generate multiple reference summaries from indirect ground truth present in VISIOCITY. We show that these summaries are at par with human summaries. We also present a study of different desired characteristics of a good summary and demonstrate how it is normal to have two good summaries with different characteristics. Thus we argue that evaluating a summary against one or more human summaries and using a single measure has its shortcomings. We propose an evaluation framework for better quantitative assessment of summary quality which is closer to human judgment. Lastly, we present insights into how a model can be enhanced to yield better summaries. Sepcifically, when multiple diverse ground truth summaries can exist, learning from them individually and using a combination of loss functions measuring different characteristics is better than learning from a single combined (oracle) ground truth summary using a single loss function. We demonstrate the effectiveness of doing so as compared to some of the representative state of the art techniques tested on VISIOCITY. We release VISIOCITY as a benchmarking dataset and invite researchers to test the effectiveness of their video summarization algorithms on VISIOCITY.


Realistic Video Summarization through VISIOCITY: A New Benchmark and Evaluation Framework

Automatic video summarization is still an unsolved problem due to severa...

A Framework towards Domain Specific Video Summarization

In the light of exponentially increasing video content, video summarizat...

PORTRAIT: a hybrid aPproach tO cReate extractive ground-TRuth summAry for dIsaster evenT

Disaster summarization approaches provide an overview of the important i...

Visual Summarization of Lecture Video Segments for Enhanced Navigation

Lecture videos are an increasingly important learning resource for highe...

Rethinking the Evaluation of Video Summaries

Video summarization is a technique to create a short skim of the origina...

Viewpoint-aware Video Summarization

This paper introduces a novel variant of video summarization, namely bui...

Summarizing the performances of a background subtraction algorithm measured on several videos

There exist many background subtraction algorithms to detect motion in v...

Please sign up or login with your details

Forgot password? Click here to reset