Relational Long Short-Term Memory for Video Action Recognition

by   Zexi Chen, et al.

Spatial and temporal relationships, both short-range and long-range, between objects in videos are key cues for recognizing actions. It is a challenging problem to model them jointly. In this paper, we first present a new variant of Long Short-Term Memory, namely Relational LSTM to address the challenge for relation reasoning across space and time between objects. In our Relational LSTM module, we utilize a non-local operation similar in spirit to the recently proposed non-local network to substitute the fully connected operation in the vanilla LSTM. By doing this, our Relational LSTM is capable of capturing long and short-range spatio-temporal relations between objects in videos in a principled way. Then, we propose a two-branch neural architecture consisting of the Relational LSTM module as the non-local branch and a spatio-temporal pooling based local branch. The local branch is introduced for capturing local spatial appearance and/or short-term motion features. The two-branch modules are concatenated to learn video-level features from snippet-level ones end-to-end. Experimental results on UCF-101 and HMDB-51 datasets show that our model achieves state-of-the-art results among LSTM-based methods, while obtaining comparable performance with other state-of-the-art methods (which use not directly comparable schema). Our code will be released.


page 6

page 7


Learning to Detect Violent Videos using Convolutional Long Short-Term Memory

Developing a technique for the automatic analysis of surveillance videos...

Convolutional Long Short-Term Memory Networks for Recognizing First Person Interactions

In this paper, we present a novel deep learning based approach for addre...

PNL: Efficient Long-Range Dependencies Extraction with Pyramid Non-Local Module for Action Recognition

Long-range spatiotemporal dependencies capturing plays an essential role...

Cubic LSTMs for Video Prediction

Predicting future frames in videos has become a promising direction of r...

Long-short Term Motion Feature for Action Classification and Retrieval

We propose a method for representing motion information for video classi...

Asynchronous Temporal Fields for Action Recognition

Actions are more than just movements and trajectories: we cook to eat an...

LSTM stack-based Neural Multi-sequence Alignment TeCHnique (NeuMATCH)

The alignment of heterogeneous sequential data (video to text) is an imp...

Please sign up or login with your details

Forgot password? Click here to reset