Multimodal Fusion Refiner Networks

by   Sethuraman Sankaran, et al.

Tasks that rely on multi-modal information typically include a fusion module that combines information from different modalities. In this work, we develop a Refiner Fusion Network (ReFNet) that enables fusion modules to combine strong unimodal representation with strong multimodal representations. ReFNet combines the fusion network with a decoding/defusing module, which imposes a modality-centric responsibility condition. This approach addresses a big gap in existing multimodal fusion frameworks by ensuring that both unimodal and fused representations are strongly encoded in the latent fusion space. We demonstrate that the Refiner Fusion Network can improve upon performance of powerful baseline fusion modules such as multimodal transformers. The refiner network enables inducing graphical representations of the fused embeddings in the latent space, which we prove under certain conditions and is supported by strong empirical results in the numerical experiments. These graph structures are further strengthened by combining the ReFNet with a Multi-Similarity contrastive loss function. The modular nature of Refiner Fusion Network lends itself to be combined with different fusion architectures easily, and in addition, the refiner step can be applied for pre-training on unlabeled datasets, thus leveraging unsupervised data towards improving performance. We demonstrate the power of Refiner Fusion Networks on three datasets, and further show that they can maintain performance with only a small fraction of labeled data.


page 1

page 6


Dynamic Fusion for Multimodal Data

Effective fusion of data from multiple modalities, such as video, speech...

Attention-guided Multi-step Fusion: A Hierarchical Fusion Network for Multimodal Recommendation

The main idea of multimodal recommendation is the rational utilization o...

A Tri-attention Fusion Guided Multi-modal Segmentation Network

In the field of multimodal segmentation, the correlation between differe...

Progressive Fusion for Multimodal Integration

Integration of multimodal information from various sources has been show...

Knowledge Fusion via Embeddings from Text, Knowledge Graphs, and Images

We present a baseline approach for cross-modal knowledge fusion. Differe...

Explaining Multimodal Data Fusion: Occlusion Analysis for Wilderness Mapping

Jointly harnessing complementary features of multi-modal input data in a...

Multimodal Representations Learning and Adversarial Hypergraph Fusion for Early Alzheimer's Disease Prediction

Multimodal neuroimage can provide complementary information about the de...

Please sign up or login with your details

Forgot password? Click here to reset