Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation

by   Shuaicheng Li, et al.

It has been found that temporal action proposal generation, which aims to discover the temporal action instances within the range of the start and end frames in the untrimmed videos, can largely benefit from proper temporal and semantic context exploitation. The latest efforts were dedicated to considering the temporal context and similarity-based semantic contexts through self-attention modules. However, they still suffer from cluttered background information and limited contextual feature learning. In this paper, we propose a novel Pyramid Region-based Slot Attention (PRSlot) module to address these issues. Instead of using the similarity computation, our PRSlot module directly learns the local relations in an encoder-decoder manner and generates the representation of a local region enhanced based on the attention over input features called slot. Specifically, upon the input snippet-level features, PRSlot module takes the target snippet as query, its surrounding region as key and then generates slot representations for each query-key slot by aggregating the local snippet context with a parallel pyramid strategy. Based on PRSlot modules, we present a novel Pyramid Region-based Slot Attention Network termed PRSA-Net to learn a unified visual representation with rich temporal and semantic context for better proposal generation. Extensive experiments are conducted on two widely adopted THUMOS14 and ActivityNet-1.3 benchmarks. Our PRSA-Net outperforms other state-of-the-art methods. In particular, we improve the AR@100 from the previous best 50.67 56.12 58.7% for action detection on THUMOS14. Code is available at <>


Accurate Temporal Action Proposal Generation with Relation-Aware Pyramid Network

Accurate temporal action proposals play an important role in detecting a...

Relaxed Transformer Decoders for Direct Action Proposal Generation

Temporal action proposal generation is an important and challenging task...

Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation

Temporal action proposal generation (TAPG) is a fundamental and challeng...

Cascaded Pyramid Mining Network for Weakly Supervised Temporal Action Localization

Weakly supervised temporal action localization, which aims at temporally...

AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation

Temporal action proposal generation (TAPG) is a challenging task, which ...

USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval

As a fundamental and challenging task in bridging language and vision do...

GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot Attention for Vision-and-Language Navigation

Most existing works solving Room-to-Room VLN problem only utilize RGB im...

Please sign up or login with your details

Forgot password? Click here to reset