Knowledge-infused Contrastive Learning for Urban Imagery-based Socioeconomic Prediction

by   Yu Liu, et al.
Helsingin yliopisto
Tsinghua University
NetEase, Inc

Monitoring sustainable development goals requires accurate and timely socioeconomic statistics, while ubiquitous and frequently-updated urban imagery in web like satellite/street view images has emerged as an important source for socioeconomic prediction. Especially, recent studies turn to self-supervised contrastive learning with manually designed similarity metrics for urban imagery representation learning and further socioeconomic prediction, which however suffers from effectiveness and robustness issues. To address such issues, in this paper, we propose a Knowledge-infused Contrastive Learning (KnowCL) model for urban imagery-based socioeconomic prediction. Specifically, we firstly introduce knowledge graph (KG) to effectively model the urban knowledge in spatiality, mobility, etc., and then build neural network based encoders to learn representations of an urban image in associated semantic and visual spaces, respectively. Finally, we design a cross-modality based contrastive learning framework with a novel image-KG contrastive loss, which maximizes the mutual information between semantic and visual representations for knowledge infusion. Extensive experiments of applying the learnt visual representations for socioeconomic prediction on three datasets demonstrate the superior performance of KnowCL with over 30% improvements on R^2 compared with baselines. Especially, our proposed KnowCL model can apply to both satellite and street imagery with both effectiveness and transferability achieved, which provides insights into urban imagery-based socioeconomic prediction.


page 1

page 7


Fair contrastive pre-training for geographic images

Contrastive representation learning is widely employed in visual recogni...

The 'Paris-end' of town? Urban typology through machine learning

The confluence of recent advances in availability of geospatial informat...

Self-Supervision, Remote Sensing and Abstraction: Representation Learning Across 3 Million Locations

Self-supervision based deep learning classification approaches have rece...

Urban2Vec: Incorporating Street View Imagery and POIs for Multi-Modal Urban Neighborhood Embedding

Understanding intrinsic patterns and predicting spatiotemporal character...

Knowledge-driven Site Selection via Urban Knowledge Graph

Site selection determines optimal locations for new stores, which is of ...

Rule based End-to-End Learning Framework for Urban Growth Prediction

Due to the rapid growth of urban areas in the past decades, it has becom...

Please sign up or login with your details

Forgot password? Click here to reset