EvHandPose: Event-based 3D Hand Pose Estimation with Sparse Supervision

by   Jianping Jiang, et al.

Event camera shows great potential in 3D hand pose estimation, especially addressing the challenges of fast motion and high dynamic range in a low-power way. However, due to the asynchronous differential imaging mechanism, it is challenging to design event representation to encode hand motion information especially when the hands are not moving (causing motion ambiguity), and it is infeasible to fully annotate the temporally dense event stream. In this paper, we propose EvHandPose with novel hand flow representations in Event-to-Pose module for accurate hand pose estimation and alleviating the motion ambiguity issue. To solve the problem under sparse annotation, we design contrast maximization and edge constraints in Pose-to-IWE (Image with Warped Events) module and formulate EvHandPose in a self-supervision framework. We further build EvRealHands, the first large-scale real-world event-based hand pose dataset on several challenging scenes to bridge the domain gap due to relying on synthetic data and facilitate future research. Experiments on EvRealHands demonstrate that EvHandPose outperforms previous event-based method under all evaluation scenes with 15 ∼ 20 mm lower MPJPE and achieves accurate and stable hand pose estimation in fast motion and strong light scenes compared with RGB-based methods. Furthermore, EvHandPose demonstrates 3D hand pose estimation at 120 fps or higher.


page 2

page 4

page 5

page 7

page 9

page 11

page 12

page 14


Lifting Monocular Events to 3D Human Poses

This paper presents a novel 3D human pose estimation approach using a si...

3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal

Estimating 3D interacting hand pose from a single RGB image is essential...

Tracking Fast by Learning Slow: An Event-based Speed Adaptive Hand Tracker Leveraging Knowledge in RGB Domain

3D hand tracking methods based on monocular RGB videos are easily affect...

Event Camera-based Visual Odometry for Dynamic Motion Tracking of a Legged Robot Using Adaptive Time Surface

Our paper proposes a direct sparse visual odometry method that combines ...

Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos

Understanding dynamic hand motions and actions from egocentric RGB video...

FetusMap: Fetal Pose Estimation in 3D Ultrasound

The 3D ultrasound (US) entrance inspires a multitude of automated prenat...

Deformable Neural Radiance Fields using RGB and Event Cameras

Modeling Neural Radiance Fields for fast-moving deformable objects from ...

Please sign up or login with your details

Forgot password? Click here to reset