Learning a Sensor-invariant Embedding of Satellite Data: A Case Study for Lake Ice Monitoring

by   Manu Tom, et al.

Fusing satellite imagery acquired with different sensors has been a long-standing challenge of Earth observation, particularly across different modalities such as optical and Synthetic Aperture Radar (SAR) images. Here, we explore the joint analysis of imagery from different sensors in the light of representation learning: we propose to learn a joint, sensor-invariant embedding (feature representation) within a deep neural network. Our application problem is the monitoring of lake ice on Alpine lakes. To reach the temporal resolution requirement of the Swiss Global Climate Observing System (GCOS) office, we combine three image sources: Sentinel-1 SAR (S1-SAR), Terra MODIS and Suomi-NPP VIIRS. The large gaps between the optical and SAR domains and between the sensor resolutions make this a challenging instance of the sensor fusion problem. Our approach can be classified as a feature-level fusion that is learnt in a data-driven manner. The proposed network architecture has separate encoding branches for each image sensor, which feed into a single latent embedding. I.e., a common feature representation shared by all inputs, such that subsequent processing steps deliver comparable output irrespective of which sort of input image was used. By fusing satellite data, we map lake ice at a temporal resolution of <1.5 days. The network produces spatially explicit lake ice maps with pixel-wise accuracies >91.3 >60.7 sets a new state-of-the-art for determining the important ice-on and ice-off dates for the target lakes, in many cases meeting the GCOS requirement.


page 1

page 5


CVPR MultiEarth 2023 Deforestation Estimation Challenge:SpaceVision4Amazon

In this paper, we present a deforestation estimation method based on att...

Land Use Prediction using Electro-Optical to SAR Few-Shot Transfer Learning

Satellite image analysis has important implications for land use, urbani...

Ice Monitoring in Swiss Lakes from Optical Satellites and Webcams using Machine Learning

Continuous observation of climate indicators, such as trends in lake fre...

Self-supervised Vision Transformers for Joint SAR-optical Representation Learning

Self-supervised learning (SSL) has attracted much interest in remote sen...

Fusing Optical and SAR time series for LAI gap filling with multioutput Gaussian processes

The availability of satellite optical information is often hampered by t...

Beyond Spatial Auto-Regressive Models: Predicting Housing Prices with Satellite Imagery

When modeling geo-spatial data, it is critical to capture spatial correl...

Please sign up or login with your details

Forgot password? Click here to reset