DeepAI AI Chat
Log In Sign Up

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

by   Solon Barocas, et al.

Several pieces of work have uncovered performance disparities by conducting "disaggregated evaluations" of AI systems. We build on these efforts by focusing on the choices that must be made when designing a disaggregated evaluation, as well as some of the key considerations that underlie these design choices and the tradeoffs between these considerations. We argue that a deeper understanding of the choices, considerations, and tradeoffs involved in designing disaggregated evaluations will better enable researchers, practitioners, and the public to understand the ways in which AI systems may be underperforming for particular groups of people.


page 1

page 2

page 3

page 4


Towards Human-centered Explainable AI: User Studies for Model Explanations

Explainable AI (XAI) is widely viewed as a sine qua non for ever-expandi...

Making Data Tangible: A Cross-disciplinary Design Space for Data Physicalization

Designing a data physicalization requires a myriad of different consider...

Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support

Various tools and practices have been developed to support practitioners...

Designing for Affective Augmentation: Assistive, Harmful, or Unfamiliar?

In what capacity are affective augmentations helpful to humans, and what...

Ethics Sheets for AI Tasks

Several high-profile events, such as the use of biased recidivism system...

Evaluating Understanding on Conceptual Abstraction Benchmarks

A long-held objective in AI is to build systems that understand concepts...

Designing a realistic peer-like embodied conversational agent for supporting children storytelling

Advances in artificial intelligence have facilitated the use of large la...