Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus

01/24/2022

∙

The punctuation restoration task aims to correctly punctuate the output transcriptions of automatic speech recognition systems. Previous punctuation models, either using text only or demanding the corresponding audio, tend to be constrained by real scenes, where unpunctuated sentences are a mixture of those with and without audio. This paper proposes a unified multimodal punctuation restoration framework, named UniPunc, to punctuate the mixed sentences with a single model. UniPunc jointly represents audio and non-audio samples in a shared latent space, based on which the model learns a hybrid representation and punctuates both kinds of samples. We validate the effectiveness of the UniPunc on real-world datasets, which outperforms various strong baselines (e.g. BERT, MuSe) by at least 0.8 overall F1 scores, making a new state-of-the-art. Extensive experiments show that UniPunc's design is a pervasive solution: by grafting onto previous models, UniPunc enables them to punctuate on the mixed corpus. Our code is available at github.com/Yaoming95/UniPunc

READ FULL TEXT

Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus

Efficient Ensemble Architecture for Multimodal Acoustic and Textual Embeddings in Punctuation Restoration using Time-Delay Neural Networks

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning

Versatile Audio-Visual Learning for Handling Single and Multi Modalities in Emotion Regression and Classification Tasks

Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations

Multimodal Punctuation Prediction with Contextual Dropout

Text-based classification of interviews for mental health – juxtaposing the state of the art

Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus

Related Research

Efficient Ensemble Architecture for Multimodal Acoustic and Textual Embeddings in Punctuation Restoration using Time-Delay Neural Networks

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning

Versatile Audio-Visual Learning for Handling Single and Multi Modalities in Emotion Regression and Classification Tasks

Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations

Multimodal Punctuation Prediction with Contextual Dropout

Text-based classification of interviews for mental health – juxtaposing the state of the art