Towards Better Fairness-Utility Trade-off: A Comprehensive Measurement-Based Reinforcement Learning Framework

by   Simiao Zhang, et al.

Machine learning is widely used to make decisions with societal impact such as bank loan approving, criminal sentencing, and resume filtering. How to ensure its fairness while maintaining utility is a challenging but crucial issue. Fairness is a complex and context-dependent concept with over 70 different measurement metrics. Since existing regulations are often vague in terms of which metric to use and different organizations may prefer different fairness metrics, it is important to have means of improving fairness comprehensively. Existing mitigation techniques often target at one specific fairness metric and have limitations in improving multiple notions of fairness simultaneously. In this work, we propose CFU (Comprehensive Fairness-Utility), a reinforcement learning-based framework, to efficiently improve the fairness-utility trade-off in machine learning classifiers. A comprehensive measurement that can simultaneously consider multiple fairness notions as well as utility is established, and new metrics are proposed based on an in-depth analysis of the relationship between different fairness metrics. The reward function of CFU is constructed with comprehensive measurement and new metrics. We conduct extensive experiments to evaluate CFU on 6 tasks, 3 machine learning models, and 15 fairness-utility measurements. The results demonstrate that CFU can improve the classifier on multiple fairness metrics without sacrificing its utility. It outperforms all state-of-the-art techniques and has witnessed a 37.5


page 1

page 8

page 9

page 10


Do the Machine Learning Models on a Crowd Sourced Platform Exhibit Bias? An Empirical Study on Model Fairness

Machine learning models are increasingly being used in important decisio...

Fairness Notions in DAG-based DLTs

This paper investigates the issue of fairness in Distributed Ledger Tech...

Copying Machine Learning Classifiers

We study model-agnostic copies of machine learning classifiers. We devel...

Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Twitter uses machine learning to crop images, where crops are centered a...

Multi-Dimensional Ability Diagnosis for Machine Learning Algorithms

Machine learning algorithms have become ubiquitous in a number of applic...

Individual Fairness Revisited: Transferring Techniques from Adversarial Robustness

We turn the definition of individual fairness on its head—rather than as...

Minimax AUC Fairness: Efficient Algorithm with Provable Convergence

The use of machine learning models in consequential decision making ofte...

Please sign up or login with your details

Forgot password? Click here to reset