Comparing Human and Machine Bias in Face Recognition

by   Samuel Dooley, et al.

Much recent research has uncovered and discussed serious concerns of bias in facial analysis technologies, finding performance disparities between groups of people based on perceived gender, skin type, lighting condition, etc. These audits are immensely important and successful at measuring algorithmic bias but have two major challenges: the audits (1) use facial recognition datasets which lack quality metadata, like LFW and CelebA, and (2) do not compare their observed algorithmic bias to the biases of their human alternatives. In this paper, we release improvements to the LFW and CelebA datasets which will enable future researchers to obtain measurements of algorithmic bias that are not tainted by major flaws in the dataset (e.g. identical images appearing in both the gallery and test set). We also use these new data to develop a series of challenging facial identification and verification questions that we administered to various algorithms and a large, balanced sample of human reviewers. We find that both computer models and human survey participants perform significantly better at the verification task, generally obtain lower accuracy rates on dark-skinned or female subjects for both tasks, and obtain higher accuracy rates when their demographics match that of the question. Computer models are observed to achieve a higher level of accuracy than the survey participants on both tasks and exhibit bias to similar degrees as the human survey participants.


page 4

page 6


Towards causal benchmarking of bias in face analysis algorithms

Measuring algorithmic bias is crucial both to assess algorithmic fairnes...

Measuring Hidden Bias within Face Recognition via Racial Phenotypes

Recent work reports disparate performance for intersectional racial grou...

FairFace Challenge at ECCV 2020: Analyzing Bias in Face Recognition

This work summarizes the 2020 ChaLearn Looking at People Fair Face Recog...

Face Recognition Techniques: A Survey

Nowadays research has expanded to extracting auxiliary information from ...

Gendered Differences in Face Recognition Accuracy Explained by Hairstyles, Makeup, and Facial Morphology

Media reports have accused face recognition of being ”biased”, ”sexist” ...

Robustness Disparities in Face Detection

Facial analysis systems have been deployed by large companies and critiq...

Understanding bias in facial recognition technologies

Over the past couple of years, the growing debate around automated facia...

Please sign up or login with your details

Forgot password? Click here to reset