Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

03/10/2021
by   Solon Barocas, et al.
5

Several pieces of work have uncovered performance disparities by conducting "disaggregated evaluations" of AI systems. We build on these efforts by focusing on the choices that must be made when designing a disaggregated evaluation, as well as some of the key considerations that underlie these design choices and the tradeoffs between these considerations. We argue that a deeper understanding of the choices, considerations, and tradeoffs involved in designing disaggregated evaluations will better enable researchers, practitioners, and the public to understand the ways in which AI systems may be underperforming for particular groups of people.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset