Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition

by   Haocong Rao, et al.

Action recognition via 3D skeleton data is an emerging important topic in these years. Most existing methods either extract hand-crafted descriptors or learn action representations by supervised learning paradigms that require massive labeled data. In this paper, we for the first time propose a contrastive action learning paradigm named AS-CAL that can leverage different augmentations of unlabeled skeleton data to learn action representations in an unsupervised manner. Specifically, we first propose to contrast similarity between augmented instances (query and key) of the input skeleton sequence, which are transformed by multiple novel augmentation strategies, to learn inherent action patterns ("pattern-invariance") of different skeleton transformations. Second, to encourage learning the pattern-invariance with more consistent action representations, we propose a momentum LSTM, which is implemented as the momentum-based moving average of LSTM based query encoder, to encode long-term action dynamics of the key sequence. Third, we introduce a queue to store the encoded keys, which allows our model to flexibly reuse proceeding keys and build a more consistent dictionary to improve contrastive learning. Last, by temporally averaging the hidden states of action learned by the query encoder, a novel representation named Contrastive Action Encoding (CAE) is proposed to represent human's action effectively. Extensive experiments show that our approach typically improves existing hand-crafted methods by 10-50 superior performance to numerous supervised learning methods.


page 1

page 9

page 10

page 11

page 16

page 17


Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning

Skeleton-based human action recognition has attracted increasing attenti...

SimMC: Simple Masked Contrastive Learning of Skeleton Representations for Unsupervised Person Re-Identification

Recent advances in skeleton-based person re-identification (re-ID) obtai...

Hierarchical Contrast for Unsupervised Skeleton-based Action Representation Learning

This paper targets unsupervised skeleton-based action representation lea...

Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentations

Contrastive learning has been proven beneficial for self-supervised skel...

VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples

MoCo is effective for unsupervised image representation learning. In thi...

Prototypical Contrast and Reverse Prediction: Unsupervised Skeleton Based Action Recognition

In this paper, we focus on unsupervised representation learning for skel...

Graph Contrastive Learning for Skeleton-based Action Recognition

In the field of skeleton-based action recognition, current top-performin...

Please sign up or login with your details

Forgot password? Click here to reset