Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks

by   Hongsong Wang, et al.

Recently, skeleton based action recognition gains more popularity due to cost-effective depth sensors coupled with real-time skeleton estimation algorithms. Traditional approaches based on handcrafted features are limited to represent the complexity of motion patterns. Recent methods that use Recurrent Neural Networks (RNN) to handle raw skeletons only focus on the contextual dependency in the temporal domain and neglect the spatial configurations of articulated skeletons. In this paper, we propose a novel two-stream RNN architecture to model both temporal dynamics and spatial configurations for skeleton based action recognition. We explore two different structures for the temporal stream: stacked RNN and hierarchical RNN. Hierarchical RNN is designed according to human body kinematics. We also propose two effective methods to model the spatial structure by converting the spatial graph into a sequence of joints. To improve generalization of our model, we further exploit 3D transformation based data augmentation techniques including rotation and scaling transformation to transform the 3D coordinates of skeletons during training. Experiments on 3D action recognition benchmark datasets show that our method brings a considerable improvement for a variety of actions, i.e., generic actions, interaction activities and gestures.


Skeleton-Based Relational Modeling for Action Recognition

With the fast development of effective and low-cost human skeleton captu...

Simultaneous Action Recognition and Human Whole-Body Motion and Dynamics Prediction from Wearable Sensors

This paper presents a novel approach to solve simultaneously the problem...

STS Classification with Dual-stream CNN

The structured time series (STS) classification problem requires the mod...

View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data

Skeleton-based human action recognition has recently attracted increasin...

Using stigmergy as a computational memory in the design of recurrent neural networks

In this paper, a novel architecture of Recurrent Neural Network (RNN) is...

Spatial-temporal Transformer-guided Diffusion based Data Augmentation for Efficient Skeleton-based Action Recognition

Recently, skeleton-based human action has become a hot research topic be...

Logsig-RNN: a novel network for robust and efficient skeleton-based action recognition

This paper contributes to the challenge of skeleton-based human action r...

Please sign up or login with your details

Forgot password? Click here to reset