Voice-Face Homogeneity Tells Deepfake

by   Harry Cheng, et al.

Detecting forgery videos is highly desirable due to the abuse of deepfake. Existing detection approaches contribute to exploring the specific artifacts in deepfake videos and fit well on certain data. However, the growing technique on these artifacts keeps challenging the robustness of traditional deepfake detectors. As a result, the development of generalizability of these approaches has reached a blockage. To address this issue, given the empirical results that the identities behind voices and faces are often mismatched in deepfake videos, and the voices and faces have homogeneity to some extent, in this paper, we propose to perform the deepfake detection from an unexplored voice-face matching view. To this end, a voice-face matching method is devised to measure the matching degree of these two. Nevertheless, training on specific deepfake datasets makes the model overfit certain traits of deepfake algorithms. We instead, advocate a method that quickly adapts to untapped forgery, with a pre-training then fine-tuning paradigm. Specifically, we first pre-train the model on a generic audio-visual dataset, followed by the fine-tuning on downstream deepfake data. We conduct extensive experiments over three widely exploited deepfake datasets - DFDC, FakeAVCeleb, and DeepfakeTIMIT. Our method obtains significant performance gains as compared to other state-of-the-art competitors. It is also worth noting that our method already achieves competitive results when fine-tuned on limited deepfake data.


page 1

page 5

page 7


SpotTune: Transfer Learning through Adaptive Fine-tuning

Transfer learning, which allows a source task to affect the inductive bi...

Self-Supervised Transformer with Domain Adaptive Reconstruction for General Face Forgery Video Detection

Face forgery videos have caused severe social public concern, and variou...

One-to-many face recognition with bilinear CNNs

The recent explosive growth in convolutional neural network (CNN) resear...

HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation

To fully leverage the advantages of large-scale pre-trained language mod...

Image-Text Pre-Training for Logo Recognition

Open-set logo recognition is commonly solved by first detecting possible...

Unmasking Deepfakes: Masked Autoencoding Spatiotemporal Transformers for Enhanced Video Forgery Detection

We present a novel approach for the detection of deepfake videos using a...

FaceOff: Anonymizing Videos in the Operating Rooms

Video capture in the surgical operating room (OR) is increasingly possib...

Please sign up or login with your details

Forgot password? Click here to reset