Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Prediction

by   Maosen Li, et al.

We propose a multiscale spatio-temporal graph neural network (MST-GNN) to predict the future 3D skeleton-based human poses in an action-category-agnostic manner. The core of MST-GNN is a multiscale spatio-temporal graph that explicitly models the relations in motions at various spatial and temporal scales. Different from many previous hierarchical structures, our multiscale spatio-temporal graph is built in a data-adaptive fashion, which captures nonphysical, yet motion-based relations. The key module of MST-GNN is a multiscale spatio-temporal graph computational unit (MST-GCU) based on the trainable graph structure. MST-GCU embeds underlying features at individual scales and then fuses features across scales to obtain a comprehensive representation. The overall architecture of MST-GNN follows an encoder-decoder framework, where the encoder consists of a sequence of MST-GCUs to learn the spatial and temporal features of motions, and the decoder uses a graph-based attention gate recurrent unit (GA-GRU) to generate future poses. Extensive experiments are conducted to show that the proposed MST-GNN outperforms state-of-the-art methods in both short and long-term motion prediction on the datasets of Human 3.6M, CMU Mocap and 3DPW, where MST-GNN outperforms previous works by 5.33 long-term prediction on Human 3.6M, and by 11.84 errors for short-term and long-term prediction on CMU Mocap, and by 1.13 mean angle errors on 3DPW in average, respectively. We further investigate the learned multiscale graphs for interpretability.


page 1

page 15


Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction

We propose novel dynamic multiscale graph neural networks (DMGNN) to pre...

Predicting Long-Term Skeletal Motions by a Spatio-Temporal Hierarchical Recurrent Network

The primary goal of skeletal motion prediction is to generate future mot...

Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs

Several applications such as autonomous driving, augmented reality and v...

Predicting Physics in Mesh-reduced Space with Temporal Attention

Graph-based next-step prediction models have recently been very successf...

Multiresolution Convolutional Autoencoders

We propose a multi-resolution convolutional autoencoder (MrCAE) architec...

DMS-GCN: Dynamic Mutiscale Spatiotemporal Graph Convolutional Networks for Human Motion Prediction

Human motion prediction is an important and challenging task in many com...

Spatio-Temporal Branching for Motion Prediction using Motion Increments

Human motion prediction (HMP) has emerged as a popular research topic du...

Please sign up or login with your details

Forgot password? Click here to reset