Recognizing American Sign Language Manual Signs from RGB-D Videos

06/07/2019
by   Longlong Jing, et al.
4

In this paper, we propose a 3D Convolutional Neural Network (3DCNN) based multi-stream framework to recognize American Sign Language (ASL) manual signs (consisting of movements of the hands, as well as non-manual face movements in some cases) in real-time from RGB-D videos, by fusing multimodality features including hand gestures, facial expressions, and body poses from multi-channels (RGB, depth, motion, and skeleton joints). To learn the overall temporal dynamics in a video, a proxy video is generated by selecting a subset of frames for each video which are then used to train the proposed 3DCNN model. We collect a new ASL dataset, ASL-100-RGBD, which contains 42 RGB-D videos captured by a Microsoft Kinect V2 camera, each of 100 ASL manual signs, including RGB channel, depth maps, skeleton joints, face features, and HDface. The dataset is fully annotated for each semantic region (i.e. the time duration of each word that the human signer performs). Our proposed method achieves 92.88 accuracy for recognizing 100 ASL words in our newly collected ASL-100-RGBD dataset. The effectiveness of our framework for recognizing hand gestures from RGB-D videos is further demonstrated on the Chalearn IsoGD dataset and achieves 76 accuracy which is 5.51 higher than the state-of-the-art work in terms of average fusion by using only 5 channels instead of 12 channels in the previous work.

READ FULL TEXT

page 2

page 5

page 6

page 7

page 9

page 10

page 11

page 12

research
05/01/2020

Recognizing American Sign Language Nonmanual Signal Grammar Errors in Continuous Videos

As part of the development of an educational tool that can help students...
research
01/07/2023

Towards early prediction of neurodevelopmental disorders: Computational model for Face Touch and Self-adaptors in Infants

Infants' neurological development is heavily influenced by their motor s...
research
04/10/2023

Isolated Sign Language Recognition based on Tree Structure Skeleton Images

Sign Language Recognition (SLR) systems aim to be embedded in video stre...
research
10/12/2021

Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble

Sign language is commonly used by deaf or mute people to communicate but...
research
09/26/2016

Lexicon-Free Fingerspelling Recognition from Video: Data, Models, and Signer Adaptation

We study the problem of recognizing video sequences of fingerspelled let...
research
09/27/2022

Video-based estimation of pain indicators in dogs

Dog owners are typically capable of recognizing behavioral cues that rev...
research
12/07/2015

Recognition from Hand Cameras

We revisit the study of a wrist-mounted camera system (referred to as Ha...

Please sign up or login with your details

Forgot password? Click here to reset