Detecting Out-of-Context Multimodal Misinformation with interpretable neural-symbolic model

by   Yizhou Zhang, et al.
University of Southern California

Recent years have witnessed the sustained evolution of misinformation that aims at manipulating public opinions. Unlike traditional rumors or fake news editors who mainly rely on generated and/or counterfeited images, text and videos, current misinformation creators now more tend to use out-of-context multimedia contents (e.g. mismatched images and captions) to deceive the public and fake news detection systems. This new type of misinformation increases the difficulty of not only detection but also clarification, because every individual modality is close enough to true information. To address this challenge, in this paper we explore how to achieve interpretable cross-modal de-contextualization detection that simultaneously identifies the mismatched pairs and the cross-modal contradictions, which is helpful for fact-check websites to document clarifications. The proposed model first symbolically disassembles the text-modality information to a set of fact queries based on the Abstract Meaning Representation of the caption and then forwards the query-image pairs into a pre-trained large vision-language model select the “evidences" that are helpful for us to detect misinformation. Extensive experiments indicate that the proposed methodology can provide us with much more interpretable predictions while maintaining the accuracy same as the state-of-the-art model on this task.


page 2

page 4


Similarity-Aware Multimodal Prompt Learning for Fake News Detection

The standard paradigm for fake news detection mainly utilizes text infor...

Multimodal Fake News Detection with Adaptive Unimodal Representation Aggregation

The development of Internet technology has continuously intensified the ...

Cross-modal Contrastive Learning for Multimodal Fake News Detection

Automatic detection of multimodal fake news has gained a widespread atte...

FakeSV: A Multimodal Benchmark with Rich Social Context for Fake News Detection on Short Video Platforms

Short video platforms have become an important channel for news sharing,...

Multimodal Fake News Detection via CLIP-Guided Learning

Multimodal fake news detection has attracted many research interests in ...

TieFake: Title-Text Similarity and Emotion-Aware Fake News Detection

Fake news detection aims to detect fake news widely spreading on social ...

Knowledge-rich Image Gist Understanding Beyond Literal Meaning

We investigate the problem of understanding the message (gist) conveyed ...

Please sign up or login with your details

Forgot password? Click here to reset