Temporal Action Localization with Multi-temporal Scales

by   Zan Gao, et al.

Temporal action localization plays an important role in video analysis, which aims to localize and classify actions in untrimmed videos. The previous methods often predict actions on a feature space of a single-temporal scale. However, the temporal features of a low-level scale lack enough semantics for action classification while a high-level scale cannot provide rich details of the action boundaries. To address this issue, we propose to predict actions on a feature space of multi-temporal scales. Specifically, we use refined feature pyramids of different scales to pass semantics from high-level scales to low-level scales. Besides, to establish the long temporal scale of the entire video, we use a spatial-temporal transformer encoder to capture the long-range dependencies of video frames. Then the refined features with long-range dependencies are fed into a classifier for the coarse action prediction. Finally, to further improve the prediction accuracy, we propose to use a frame-level self attention module to refine the classification and boundaries of each action instance. Extensive experiments show that the proposed method can outperform state-of-the-art approaches on the THUMOS14 dataset and achieves comparable performance on the ActivityNet1.3 dataset. Compared with A2Net (TIP20, Avg{0.3:0.7}), Sub-Action (CSVT2022, Avg{0.1:0.5}), and AFSD (CVPR21, Avg{0.3:0.7}) on the THUMOS14 dataset, the proposed method can achieve improvements of 12.6%, 17.4% and 2.2%, respectively


page 1

page 2

page 10


MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation

Temporally locating and classifying action segments in long untrimmed vi...

ReLER@ZJU Submission to the Ego4D Moment Queries Challenge 2022

In this report, we present the ReLER@ZJU1 submission to the Ego4D Moment...

Learning Sparse Temporal Video Mapping for Action Quality Assessment in Floor Gymnastics

Athlete performance measurement in sports videos requires modeling long ...

Video Self-Stitching Graph Network for Temporal Action Localization

Temporal action localization (TAL) in videos is a challenging task, espe...

Temporal Convolutional Networks: A Unified Approach to Action Segmentation

The dominant paradigm for video-based action segmentation is composed of...

A Study of Actor and Action Semantic Retention in Video Supervoxel Segmentation

Existing methods in the semantic computer vision community seem unable t...

Temporal Progressive Attention for Early Action Prediction

Early action prediction deals with inferring the ongoing action from par...

Please sign up or login with your details

Forgot password? Click here to reset