TIMIT-TTS: a Text-to-Speech Dataset for Multimodal Synthetic Media Detection

by   Davide Salvi, et al.

With the rapid development of deep learning techniques, the generation and counterfeiting of multimedia material are becoming increasingly straightforward to perform. At the same time, sharing fake content on the web has become so simple that malicious users can create unpleasant situations with minimal effort. Also, forged media are getting more and more complex, with manipulated videos that are taking the scene over still images. The multimedia forensic community has addressed the possible threats that this situation could imply by developing detectors that verify the authenticity of multimedia objects. However, the vast majority of these tools only analyze one modality at a time. This was not a problem as long as still images were considered the most widely edited media, but now, since manipulated videos are becoming customary, performing monomodal analyses could be reductive. Nonetheless, there is a lack in the literature regarding multimodal detectors, mainly due to the scarsity of datasets containing forged multimodal data to train and test the designed algorithms. In this paper we focus on the generation of an audio-visual deepfake dataset. First, we present a general pipeline for synthesizing speech deepfake content from a given real or fake video, facilitating the creation of counterfeit multimodal material. The proposed method uses Text-to-Speech (TTS) and Dynamic Time Warping techniques to achieve realistic speech tracks. Then, we use the pipeline to generate and release TIMIT-TTS, a synthetic speech dataset containing the most cutting-edge methods in the TTS field. This can be used as a standalone audio dataset, or combined with other state-of-the-art sets to perform multimodal research. Finally, we present numerous experiments to benchmark the proposed dataset in both mono and multimodal conditions, showing the need for multimodal forensic detectors and more suitable data.


All-for-One and One-For-All: Deep learning-based feature fusion for Synthetic Speech Detection

Recent advances in deep learning and computer vision have made the synth...

Forensic Analysis of Synthetically Generated Scientific Images

The widespread diffusion of synthetically generated content is a serious...

Media Forensics and DeepFakes: an overview

With the rapid progress of recent years, techniques that generate and ma...

Why Do Deepfake Detectors Fail?

Recent rapid advancements in deepfake technology have allowed the creati...

Evoking Places from Spaces. The application of multimodal narrative techniques in the creation of "U Modified"

Multimodal diegetic narrative tools, as applied in multimedia arts pract...

Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text

Real world multimedia data is often composed of multiple modalities such...

Towards generalizing deep-audio fake detection networks

Today's generative neural networks allow the creation of high-quality sy...

Please sign up or login with your details

Forgot password? Click here to reset