Data-Free Evaluation of User Contributions in Federated Learning

by   Hongtao Lv, et al.
Shanghai Jiao Tong University
Alibaba Group
The University of Texas at Dallas

Federated learning (FL) trains a machine learning model on mobile devices in a distributed manner using each device's private data and computing resources. A critical issues is to evaluate individual users' contributions so that (1) users' effort in model training can be compensated with proper incentives and (2) malicious and low-quality users can be detected and removed. The state-of-the-art solutions require a representative test dataset for the evaluation purpose, but such a dataset is often unavailable and hard to synthesize. In this paper, we propose a method called Pairwise Correlated Agreement (PCA) based on the idea of peer prediction to evaluate user contribution in FL without a test dataset. PCA achieves this using the statistical correlation of the model parameters uploaded by users. We then apply PCA to designing (1) a new federated learning algorithm called Fed-PCA, and (2) a new incentive mechanism that guarantees truthfulness. We evaluate the performance of PCA and Fed-PCA using the MNIST dataset and a large industrial product recommendation dataset. The results demonstrate that our Fed-PCA outperforms the canonical FedAvg algorithm and other baseline methods in accuracy, and at the same time, PCA effectively incentivizes users to behave truthfully.


page 1

page 2

page 3

page 4


Estimation of Individual Device Contributions for Incentivizing Federated Learning

Federated learning (FL) is an emerging technique used to train a machine...

A Practical Cross-Device Federated Learning Framework over 5G Networks

The concept of federated learning (FL) was first proposed by Google in 2...

User-Oriented Multi-Task Federated Deep Learning for Mobile Edge Computing

Federated Learning (FL) is a recent approach for collaboratively trainin...

FLeet: Online Federated Learning via Staleness Awareness and Performance Prediction

Federated Learning (FL) is very appealing for its privacy benefits: esse...

Equitable-FL: Federated Learning with Sparsity for Resource-Constrained Environment

In Federated Learning, model training is performed across multiple compu...

Please sign up or login with your details

Forgot password? Click here to reset