AutoFed: Heterogeneity-Aware Federated Multimodal Learning for Robust Autonomous Driving

by   Tianyue Zheng, et al.

Object detection with on-board sensors (e.g., lidar, radar, and camera) play a crucial role in autonomous driving (AD), and these sensors complement each other in modalities. While crowdsensing may potentially exploit these sensors (of huge quantity) to derive more comprehensive knowledge, federated learning (FL) appears to be the necessary tool to reach this potential: it enables autonomous vehicles (AVs) to train machine learning models without explicitly sharing raw sensory data. However, the multimodal sensors introduce various data heterogeneity across distributed AVs (e.g., label quantity skews and varied modalities), posing critical challenges to effective FL. To this end, we present AutoFed as a heterogeneity-aware FL framework to fully exploit multimodal sensory data on AVs and thus enable robust AD. Specifically, we first propose a novel model leveraging pseudo-labeling to avoid mistakenly treating unlabeled objects as the background. We also propose an autoencoder-based data imputation method to fill missing data modality (of certain AVs) with the available ones. To further reconcile the heterogeneity, we finally present a client selection mechanism exploiting the similarities among client models to improve both training stability and convergence rate. Our experiments on benchmark dataset confirm that AutoFed substantially improves over status quo approaches in both precision and recall, while demonstrating strong robustness to adverse weather conditions.


page 5

page 9

page 10

page 11


aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception

Autonomous driving is a popular research area within the computer vision...

Communication Resources Constrained Hierarchical Federated Learning for End-to-End Autonomous Driving

While federated learning (FL) improves the generalization of end-to-end ...

FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning

Federated learning (FL) enables a decentralized machine learning paradig...

Federated Learning for Vehicular Networks

Machine learning (ML) has already been adopted in vehicular networks for...

Personalized Federated Learning of Driver Prediction Models for Autonomous Driving

Autonomous vehicles (AVs) must interact with a diverse set of human driv...

A Dispersed Federated Learning Framework for 6G-Enabled Autonomous Driving Cars

Sixth-Generation (6G)-based Internet of Everything applications (e.g. au...

Please sign up or login with your details

Forgot password? Click here to reset