DeepAI AI Chat
Log In Sign Up

Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation

by   Necati Cihan Camgoz, et al.
University of Surrey

Prior work on Sign Language Translation has shown that having a mid-level sign gloss representation (effectively recognizing the individual signs) improves the translation performance drastically. In fact, the current state-of-the-art in translation requires gloss level tokenization in order to work. We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation while being trainable in an end-to-end manner. This is achieved by using a Connectionist Temporal Classification (CTC) loss to bind the recognition and translation problems into a single unified architecture. This joint approach does not require any ground-truth timing information, simultaneously solving two co-dependant sequence-to-sequence learning problems and leads to significant performance gains. We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset. We report state-of-the-art sign language recognition and translation results achieved by our Sign Language Transformers. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models, in some cases more than doubling the performance (9.58 vs. 21.80 BLEU-4 Score). We also share new baseline translation results using transformer networks for several other text-to-text sign language translation tasks.


page 1

page 2

page 3

page 4


Sign Language Translation with Transformers

Sign Language Translation (SLT) first uses a Sign Language Recognition (...

Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse

Sign Language Translation (SLT) is a challenging task that aims to gener...

Attention-Driven Multi-Modal Fusion: Enhancing Sign Language Recognition and Translation

In this paper, we devise a mechanism for the addition of multi-modal inf...

Progressive Transformers for End-to-End Sign Language Production

The goal of automatic Sign Language Production (SLP) is to translate spo...

Multi-channel Transformers for Multi-articulatory Sign Language Translation

Sign languages use multiple asynchronous information channels (articulat...

SimulSLT: End-to-End Simultaneous Sign Language Translation

Sign language translation as a kind of technology with profound social s...

Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation

Sign language recognition and translation first uses a recognition modul...