DeepAI AI Chat
Log In Sign Up

Recognizing American Sign Language Nonmanual Signal Grammar Errors in Continuous Videos

by   Elahe Vahdani, et al.
Rochester Institute of Technology
CUNY Law School

As part of the development of an educational tool that can help students achieve fluency in American Sign Language (ASL) through independent and interactive practice with immediate feedback, this paper introduces a near real-time system to recognize grammatical errors in continuous signing videos without necessarily identifying the entire sequence of signs. Our system automatically recognizes if performance of ASL sentences contains grammatical errors made by ASL students. We first recognize the ASL grammatical elements including both manual gestures and nonmanual signals independently from multiple modalities (i.e. hand gestures, facial expressions, and head movements) by 3D-ResNet networks. Then the temporal boundaries of grammatical elements from different modalities are examined to detect ASL grammatical mistakes by using a sliding window-based approach. We have collected a dataset of continuous sign language, ASL-HW-RGBD, covering different aspects of ASL grammars for training and testing. Our system is able to recognize grammatical elements on ASL-HW-RGBD from manual gestures, facial expressions, and head movements and successfully detect 8 ASL grammatical mistakes.


page 1

page 3

page 5

page 6


Recognizing American Sign Language Manual Signs from RGB-D Videos

In this paper, we propose a 3D Convolutional Neural Network (3DCNN) base...

Automated Analysis and Prediction of Job Interview Performance

We present a computational framework for automatically quantifying verba...

Neural Sign Reenactor: Deep Photorealistic Sign Language Retargeting

In this paper, we introduce a neural rendering pipeline for transferring...

ADDSL: Hand Gesture Detection and Sign Language Recognition on Annotated Danish Sign Language

For a long time, detecting hand gestures and recognizing them as letters...

Building Korean Sign Language Augmentation (KoSLA) Corpus with Data Augmentation Technique

We present an efficient framework of corpus for sign language translatio...

Global-local Enhancement Network for NMFs-aware Sign Language Recognition

Sign language recognition (SLR) is a challenging problem, involving comp...

Exploring Temporal Dependencies in Multimodal Referring Expressions with Mixed Reality

In collaborative tasks, people rely both on verbal and non-verbal cues s...