Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images

by   Pallavi Jain, et al.

Self-Supervised learning (SSL) has become the new state-of-art in several domain classification and segmentation tasks. Of these, one popular category in SSL is distillation networks such as BYOL. This work proposes RSDnet, which applies the distillation network (BYOL) in the remote sensing (RS) domain where data is non-trivially different from natural RGB images. Since Multi-spectral (MS) and synthetic aperture radar (SAR) sensors provide varied spectral and spatial resolution information, we utilised them as an implicit augmentation to learn invariant feature embeddings. In order to learn RS based invariant features with SSL, we trained RSDnet in two ways, i.e., single channel feature learning and three channel feature learning. This work explores the usefulness of single channel feature learning from random MS and SAR bands compared to the common notion of using three or more bands. In our linear evaluation, these single channel features reached a 0.92 F1 score on the EuroSAT classification task and 59.6 mIoU on the DFC segmentation task for certain single bands. We also compared our results with ImageNet weights and showed that the RS based SSL model outperforms the supervised ImageNet based model. We further explored the usefulness of multi-modal data compared to single modality data, and it is shown that utilising MS and SAR data learn better invariant representations than utilising only MS data.


page 7

page 8


MS-Net: A Multi-modal Self-supervised Network for Fine-Grained Classification of Aircraft in SAR Images

Synthetic aperture radar (SAR) imaging technology is commonly used to pr...

Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works

Deep learning has achieved great success in learning features from massi...

Multi-Modal Fusion Transformer for Visual Question Answering in Remote Sensing

With the new generation of satellite technologies, the archives of remot...

Self-supervised Vision Transformers for Joint SAR-optical Representation Learning

Self-supervised learning (SSL) has attracted much interest in remote sen...

The color out of space: learning self-supervised representations for Earth Observation imagery

The recent growth in the number of satellite images fosters the developm...

Multi-dimension Geospatial feature learning for urban region function recognition

Urban region function recognition plays a vital character in monitoring ...

Multi-Channel Auto-Encoders and a Novel Dataset for Learning Domain Invariant Representations of Histopathology Images

Domain shift is a problem commonly encountered when developing automated...

Please sign up or login with your details

Forgot password? Click here to reset