Self-supervised Audiovisual Representation Learning for Remote Sensing Data

by   Konrad Heidler, et al.

Many current deep learning approaches make extensive use of backbone networks pre-trained on large datasets like ImageNet, which are then fine-tuned to perform a certain task. In remote sensing, the lack of comparable large annotated datasets and the wide diversity of sensing platforms impedes similar developments. In order to contribute towards the availability of pre-trained backbone networks in remote sensing, we devise a self-supervised approach for pre-training deep neural networks. By exploiting the correspondence between geo-tagged audio recordings and remote sensing imagery, this is done in a completely label-free manner, eliminating the need for laborious manual annotation. For this purpose, we introduce the SoundingEarth dataset, which consists of co-located aerial imagery and audio samples all around the world. Using this dataset, we then pre-train ResNet models to map samples from both modalities into a common embedding space, which encourages the models to understand key properties of a scene that influence both visual and auditory appearance. To validate the usefulness of the proposed approach, we evaluate the transfer learning performance of pre-trained weights obtained against weights obtained through other means. By fine-tuning the models on a number of commonly used remote sensing datasets, we show that our approach outperforms existing pre-training strategies for remote sensing imagery. The dataset, code and pre-trained model weights will be available at


page 1

page 3

page 5

page 8


Self-Supervised In-Domain Representation Learning for Remote Sensing Image Scene Classification

Transferring the ImageNet pre-trained weights to the various remote sens...

The Role of Pre-Training in High-Resolution Remote Sensing Scene Classification

Due to the scarcity of labeled data, using models pre-trained on ImageNe...

Towards Better Exploiting Convolutional Neural Networks for Remote Sensing Scene Classification

We present an analysis of three possible strategies for exploiting the p...

Visual Feature Encoding for GNNs on Road Networks

In this work, we present a novel approach to learning an encoding of vis...

Scalable Label-efficient Footpath Network Generation Using Remote Sensing Data and Self-supervised Learning

Footpath mapping, modeling, and analysis can provide important geospatia...

Disentangled Pre-training for Image Matting

Image matting requires high-quality pixel-level human annotations to sup...

Active Learning Guided Fine-Tuning for enhancing Self-Supervised Based Multi-Label Classification of Remote Sensing Images

In recent years, deep neural networks (DNNs) have been found very succes...

Please sign up or login with your details

Forgot password? Click here to reset