Deep Adaptive Temporal Pooling for Activity Recognition

by   Sibo Song, et al.
Keele University
Agency for Science, Technology and Research
Singapore University of Technology and Design

Deep neural networks have recently achieved competitive accuracy for human activity recognition. However, there is room for improvement, especially in modeling long-term temporal importance and determining the activity relevance of different temporal segments in a video. To address this problem, we propose a learnable and differentiable module: Deep Adaptive Temporal Pooling (DATP). DATP applies a self-attention mechanism to adaptively pool the classification scores of different video segments. Specifically, using frame-level features, DATP regresses importance of different temporal segments and generates weights for them. Remarkably, DATP is trained using only the video-level label. There is no need of additional supervision except video-level activity class label. We conduct extensive experiments to investigate various input features and different weight models. Experimental results show that DATP can learn to assign large weights to key video segments. More importantly, DATP can improve training of frame-level feature extractor. This is because relevant temporal segments are assigned large weights during back-propagation. Overall, we achieve state-of-the-art performance on UCF101, HMDB51 and Kinetics datasets.


page 3

page 7


Multi-Label Activity Recognition using Activity-specific Features

We introduce an approach to multi-label activity recognition by extracti...

Discriminatively Learned Hierarchical Rank Pooling Networks

In this work, we present novel temporal encoding methods for action and ...

Temporal Reasoning Graph for Activity Recognition

Despite great success has been achieved in activity analysis, it still h...

Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge

Current state-of-the-art human activity recognition is focused on the cl...

A Video Summarization Method Using Temporal Interest Detection and Key Frame Prediction

In this paper, a Video Summarization Method using Temporal Interest Dete...

Spatial Parsing and Dynamic Temporal Pooling networks for Human-Object Interaction detection

The key of Human-Object Interaction(HOI) recognition is to infer the rel...

Batch-Based Activity Recognition from Egocentric Photo-Streams

Activity recognition from long unstructured egocentric photo-streams has...

Please sign up or login with your details

Forgot password? Click here to reset