Quantifying the Impact of User Attention on Fair Group Representation in Ranked Lists

by   Piotr Sapiezynski, et al.

In this work we introduce a novel metric for verifying group fairness in ranked lists. Our approach relies on measuring the amount of attention given to members of a protected group and comparing it to that group's representation in the investigated population. It offers two major developments compared to the state of the art. First, rather than assuming a logarithmic loss in importance as a function of the rank, we allow for attention distributions that are specific to the audited service and the habits of its users. For example, we expect a user to see more items during a single viewing of a social media feed than when they inspect the list of results for a single query on a web search engine. Second, we allow non-binary protected attributes to enable investigating inherently continuous attributes (for example political alignment on the Democratic vs. Republican spectrum) as well as to facilitate measurements across aggregated set of search results, rather than separately for each result list. Finally, we showcase the metric through a simulated audit of a hiring service, an online dating service, and a search engine. We show that knowing the usage patterns of the particular service is crucial in determining the fairness of its results---depending on the attention distribution function, the same list of results can appear biased both in favor and against a protected group.


page 2

page 3

page 4

page 5

page 6

page 7

page 9

page 11


A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevance

We present a simple and versatile framework for evaluating ranked lists ...

Fair Ranking with Noisy Protected Attributes

The fair-ranking problem, which asks to rank a given set of items to max...

MANI-Rank: Multiple Attribute and Intersectional Group Fairness for Consensus Ranking

Combining the preferences of many rankers into one single consensus rank...

On allocations that give intersecting groups their fair share

We consider item allocation to individual agents who have additive valua...

An Outcome Test of Discrimination for Ranked Lists

This paper extends Becker (1957)'s outcome test of discrimination to set...

A Probabilistic Position Bias Model for Short-Video Recommendation Feeds

Modern web-based platforms show ranked lists of recommendations to users...

Learning to Truncate Ranked Lists for Information Retrieval

Ranked list truncation is of critical importance in a variety of profess...

Please sign up or login with your details

Forgot password? Click here to reset