Space-time Mixing Attention for Video Transformer

by   Adrian Bulat, et al.

This paper is on video recognition using Transformers. Very recent attempts in this area have demonstrated promising results in terms of recognition accuracy, yet they have been also shown to induce, in many cases, significant computational overheads due to the additional modelling of the temporal information. In this work, we propose a Video Transformer model the complexity of which scales linearly with the number of frames in the video sequence and hence induces no overhead compared to an image-based Transformer model. To achieve this, our model makes two approximations to the full space-time attention used in Video Transformers: (a) It restricts time attention to a local temporal window and capitalizes on the Transformer's depth to obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing to attend jointly spatial and temporal locations without inducing any additional cost on top of a spatial-only attention model. We also show how to integrate 2 very lightweight mechanisms for global temporal-only attention which provide additional accuracy improvements at minimal computational cost. We demonstrate that our model produces very high recognition accuracy on the most popular video recognition datasets while at the same time being significantly more efficient than other Video Transformer models. Code will be made available.


page 1

page 2

page 3

page 4


Deformable Video Transformer

Video transformers have recently emerged as an effective alternative to ...

Efficient Attention-free Video Shift Transformers

This paper tackles the problem of efficient video recognition. In this a...

What can human minimal videos tell us about dynamic recognition models?

In human vision objects and their parts can be visually recognized from ...

Time-Space Transformers for Video Panoptic Segmentation

We propose a novel solution for the task of video panoptic segmentation,...

Dynamic Scene Video Deblurring using Non-Local Attention

This paper tackles the challenging problem of video deblurring. Most of ...

Decoupled Spatial-Temporal Transformer for Video Inpainting

Video inpainting aims to fill the given spatiotemporal holes with realis...

Optimizing ViViT Training: Time and Memory Reduction for Action Recognition

In this paper, we address the challenges posed by the substantial traini...

Please sign up or login with your details

Forgot password? Click here to reset