Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition

by   Ziyang Ma, et al.

In this paper, we explored how to boost speech emotion recognition (SER) with the state-of-the-art speech pre-trained model (PTM), data2vec, text generation technique, GPT-4, and speech synthesis technique, Azure TTS. First, we investigated the representation ability of different speech self-supervised pre-trained models, and we found that data2vec has a good representation ability on the SER task. Second, we employed a powerful large language model (LLM), GPT-4, and emotional text-to-speech (TTS) model, Azure TTS, to generate emotionally congruent text and speech. We carefully designed the text prompt and dataset construction, to obtain the synthetic emotional speech data with high quality. Third, we studied different ways of data augmentation to promote the SER task with synthetic speech, including random mixing, adversarial training, transfer learning, and curriculum learning. Experiments and ablation studies on the IEMOCAP dataset demonstrate the effectiveness of our method, compared with other data augmentation methods, and data augmentation with other synthetic data.


page 1

page 2

page 3

page 4


Generative Emotional AI for Speech Emotion Recognition: The Case for Synthetic Emotional Speech Augmentation

Despite advances in deep learning, current state-of-the-art speech emoti...

A Preliminary Study on Augmenting Speech Emotion Recognition using a Diffusion Model

In this paper, we propose to utilise diffusion models for data augmentat...

Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation

In Speech Emotion Recognition (SER), emotional characteristics often app...

SpeechBlender: Speech Augmentation Framework for Mispronunciation Data Generation

One of the biggest challenges in designing mispronunciation detection mo...

Noisy Self-Training with Data Augmentations for Offensive and Hate Speech Detection Tasks

Online social media is rife with offensive and hateful comments, prompti...

Gaussian-smoothed Imbalance Data Improves Speech Emotion Recognition

In speech emotion recognition tasks, models learn emotional representati...

Data Augmentation with Unsupervised Speaking Style Transfer for Speech Emotion Recognition

Currently, the performance of Speech Emotion Recognition (SER) systems i...

Please sign up or login with your details

Forgot password? Click here to reset