Responsible Scoring Mechanisms Through Function Sampling

by   Abolfazl Asudeh, et al.

Human decision-makers often receive assistance from data-driven algorithmic systems that provide a score for evaluating objects, including individuals. The scores are generated by a function (mechanism) that takes a set of features as input and generates a score.The scoring functions are either machine-learned or human-designed and can be used for different decision purposes such as ranking or classification. Given the potential impact of these scoring mechanisms on individuals' lives and on society, it is important to make sure these scores are computed responsibly. Hence we need tools for responsible scoring mechanism design. In this paper, focusing on linear scoring functions, we highlight the importance of unbiased function sampling and perturbation in the function space for devising such tools. We provide unbiased samplers for the entire function space, as well as a θ-vicinity around a given function. We then illustrate the value of these samplers for designing effective algorithms in three diverse problem scenarios in the context of ranking. Finally, as a fundamental method for designing responsible scoring mechanisms, we propose a novel approach for approximating the construction of the arrangement of hyperplanes. Despite the exponential complexity of an arrangement in the number of dimensions, using function sampling, our algorithm is linear in the number of samples and hyperplanes, and independent of the number of dimensions.


Learning Fair Scoring Functions: Fairness Definitions, Algorithms and Generalization Bounds for Bipartite Ranking

Many applications of artificial intelligence, ranging from credit lendin...

Analysis of Multivariate Scoring Functions for Automatic Unbiased Learning to Rank

Leveraging biased click data for optimizing learning to rank systems has...

Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations, and Forecast Rankings

In the practice of point prediction, it is desirable that forecasters re...

Maximizing Welfare with Incentive-Aware Evaluation Mechanisms

Motivated by applications such as college admission and insurance rate d...

Linear Classifier Combination via Multiple Potential Functions

A vital aspect of the classification based model construction process is...

A Family of Metrics for Clustering Algorithms

We give the motivation for scoring clustering algorithms and a metric M ...

Prudence When Assuming Normality: an advice for machine learning practitioners

In a binary classification problem the feature vector (predictor) is the...

Please sign up or login with your details

Forgot password? Click here to reset