"What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairness

by   McKane Andrus, et al.

As calls for fair and unbiased algorithmic systems increase, so too does the number of individuals working on algorithmic fairness in industry. However, these practitioners often do not have access to the demographic data they feel they need to detect bias in practice. Even with the growing variety of toolkits and strategies for working towards algorithmic fairness, they almost invariably require access to demographic attributes or proxies. We investigated this dilemma through semi-structured interviews with 38 practitioners and professionals either working in or adjacent to algorithmic fairness. Participants painted a complex picture of what demographic data availability and use look like on the ground, ranging from not having access to personal data of any kind to being legally required to collect and use demographic data for discrimination assessments. In many domains, demographic data collection raises a host of difficult questions, including how to balance privacy and fairness, how to define relevant social categories, how to ensure meaningful consent, and whether it is appropriate for private companies to infer someone's demographics. Our research suggests challenges that must be considered by businesses, regulators, researchers, and community groups in order to enable practitioners to address algorithmic bias in practice. Critically, we do not propose that the overall goal of future work should be to simply lower the barriers to collecting demographic data. Rather, our study surfaces a swath of normative questions about how, when, and even whether this data should be collected.


page 1

page 2

page 3

page 4


Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection in the Pursuit of Fairness

Most proposed algorithmic fairness techniques require access to data on ...

Awareness in Practice: Tensions in Access to Sensitive Attribute Data for Antidiscrimination

Organizations cannot address demographic disparities that they cannot se...

Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness

Developing robust and fair AI systems require datasets with comprehensiv...

Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support

Various tools and practices have been developed to support practitioners...

Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems

Many technical approaches have been proposed for ensuring that decisions...

Improving Fairness in Large-Scale Object Recognition by CrowdSourced Demographic Information

There has been increasing awareness of ethical issues in machine learnin...

Assessing and Addressing Algorithmic Bias - But Before We Get There

Algorithmic and data bias are gaining attention as a pressing issue in p...

Please sign up or login with your details

Forgot password? Click here to reset