Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data

by   Yuxiao Zhou, et al.

We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps and at state-of-the-art accuracy. This is enabled by a new learning based architecture designed such that it can make use of all the sources of available hand training data: image data with either 2D or 3D annotations, as well as stand-alone 3D animations without corresponding image data. It features a 3D hand joint detection module and an inverse kinematics module which regresses not only 3D joint positions but also maps them to joint rotations in a single feed-forward pass. This output makes the method more directly usable for applications in computer vision and graphics compared to only regressing 3D joint positions. We demonstrate that our architectural design leads to a significant quantitative and qualitative improvement over the state of the art on several challenging benchmarks. Our model is publicly available for future research.


page 1

page 6

page 8


HandTailor: Towards High-Precision Monocular 3D Hand Recovery

3D hand pose estimation and shape recovery are challenging tasks in comp...

DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth

Articulated hand pose and shape estimation is an important problem for v...

Using a single RGB frame for real time 3D hand pose estimation in the wild

We present a method for the real-time estimation of the full 3D pose of ...

Denoising Diffusion for 3D Hand Pose Estimation from Images

Hand pose estimation from a single image has many applications. However,...

MM-Hand: 3D-Aware Multi-Modal Guided Hand Generative Network for 3D Hand Pose Synthesis

Estimating the 3D hand pose from a monocular RGB image is important but ...

BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks

3D hand estimation has been a long-standing research topic in computer v...

Creatures great and SMAL: Recovering the shape and motion of animals from video

We present a system to recover the 3D shape and motion of a wide variety...

Please sign up or login with your details

Forgot password? Click here to reset