Cross-Modal Knowledge Transfer Without Task-Relevant Source Data

by   Sk. Miraj Ahmed, et al.

Cost-effective depth and infrared sensors as alternatives to usual RGB sensors are now a reality, and have some advantages over RGB in domains like autonomous navigation and remote sensing. As such, building computer vision and deep learning systems for depth and infrared data are crucial. However, large labeled datasets for these modalities are still lacking. In such cases, transferring knowledge from a neural network trained on a well-labeled large dataset in the source modality (RGB) to a neural network that works on a target modality (depth, infrared, etc.) is of great value. For reasons like memory and privacy, it may not be possible to access the source data, and knowledge transfer needs to work with only the source models. We describe an effective solution, SOCKET: SOurce-free Cross-modal KnowledgE Transfer for this challenging task of transferring knowledge from one source modality to a different target modality without access to task-relevant source data. The framework reduces the modality gap using paired task-irrelevant data, as well as by matching the mean and variance of the target features with the batch-norm statistics that are present in the source models. We show through extensive experiments that our method significantly outperforms existing source-free methods for classification tasks which do not account for the modality gap.


page 19

page 20


Feature-Supervised Action Modality Transfer

This paper strives for action recognition and detection in video modalit...

A Cross-Modal Distillation Network for Person Re-identification in RGB-Depth

Person re-identification involves the recognition over time of individua...

Towards Privacy-Supporting Fall Detection via Deep Unsupervised RGB2Depth Adaptation

Fall detection is a vital task in health monitoring, as it allows the sy...

Towards All-around Knowledge Transferring: Learning From Task-irrelevant Labels

Deep neural models have hitherto achieved significant performances on nu...

Low to High Dimensional Modality Hallucination using Aggregated Fields of View

Real-world robotics systems deal with data from a multitude of modalitie...

Differentiable Weight Masks for Domain Transfer

One of the major drawbacks of deep learning models for computer vision h...

Probabilistic Knowledge Transfer for Deep Representation Learning

Knowledge Transfer (KT) techniques tackle the problem of transferring th...

Please sign up or login with your details

Forgot password? Click here to reset