Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection

by   Mengzhu Sun, et al.

Rumor spreaders are increasingly utilizing multimedia content to attract the attention and trust of news consumers. Though quite a few rumor detection models have exploited the multi-modal data, they seldom consider the inconsistent semantics between images and texts, and rarely spot the inconsistency among the post contents and background knowledge. In addition, they commonly assume the completeness of multiple modalities and thus are incapable of handling handle missing modalities in real-life scenarios. Motivated by the intuition that rumors in social media are more likely to have inconsistent semantics, a novel Knowledge-guided Dual-consistency Network is proposed to detect rumors with multimedia contents. It uses two consistency detection subnetworks to capture the inconsistency at the cross-modal level and the content-knowledge level simultaneously. It also enables robust multi-modal representation learning under different missing visual modality conditions, using a special token to discriminate between posts with visual modality and posts without visual modality. Extensive experiments on three public real-world multimedia datasets demonstrate that our framework can outperform the state-of-the-art baselines under both complete and incomplete modality conditions. Our codes are available at


page 2

page 4

page 11

page 14


Flexible-modal Deception Detection with Audio-Visual Adapter

Detecting deception by human behaviors is vital in many fields such as c...

Multi-Modal Semantic Inconsistency Detection in Social Media News Posts

As computer-generated content and deepfakes make steady improvements, se...

Deep Structured Cross-Modal Anomaly Detection

Anomaly detection is a fundamental problem in data mining field with man...

Heri-Graphs: A Workflow of Creating Datasets for Multi-modal Machine Learning on Graphs of Heritage Values and Attributes with Social Media

Values (why to conserve) and Attributes (what to conserve) are essential...

On the Limits to Multi-Modal Popularity Prediction on Instagram – A New Robust, Efficient and Explainable Baseline

The predictability of social media popularity is a topic of much scienti...

COVID-VTS: Fact Extraction and Verification on Short Video Platforms

We introduce a new benchmark, COVID-VTS, for fact-checking multi-modal i...

Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understanding

Transformers achieve promising performance in document understanding bec...

Please sign up or login with your details

Forgot password? Click here to reset