Unsupervised Opinion Aggregation – A Statistical Perspective

08/20/2023
by   Noyan C. Sevuktekin, et al.
0

Complex decision-making systems rarely have direct access to the current state of the world and they instead rely on opinions to form an understanding of what the ground truth could be. Even in problems where experts provide opinions without any intention to manipulate the decision maker, it is challenging to decide which expert's opinion is more reliable – a challenge that is further amplified when decision-maker has limited, delayed, or no access to the ground truth after the fact. This paper explores a statistical approach to infer the competence of each expert based on their opinions without any need for the ground truth. Echoing the logic behind what is commonly referred to as the wisdom of crowds, we propose measuring the competence of each expert by their likeliness to agree with their peers. We further show that the more reliable an expert is the more likely it is that they agree with their peers. We leverage this fact to propose a completely unsupervised version of the naïve Bayes classifier and show that the proposed technique is asymptotically optimal for a large class of problems. In addition to aggregating a large block of opinions, we further apply our technique for online opinion aggregation and for decision-making based on a limited the number of opinions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/04/2021

Utilizing Expert Opinion to inform Extrapolation of Survival Models

In decision modelling with time to event data, there are a variety of pa...
research
05/15/2023

MADDM: Multi-Advisor Dynamic Binary Decision-Making by Maximizing the Utility

Being able to infer ground truth from the responses of multiple imperfec...
research
07/18/2023

Conformal prediction under ambiguous ground truth

In safety-critical classification tasks, conformal prediction allows to ...
research
03/26/2017

Who Said What: Modeling Individual Labelers Improves Classification

Data are often labeled by many different experts with each expert only l...
research
07/25/2022

Optimal Prior Pooling from Expert Opinions

The pooling of prior opinions is an important area of research and has b...
research
05/28/2021

The Generation of Security Scoring Systems Leveraging Human Expert Opinion

While the existence of many security elements can be measured (e.g., vul...
research
10/12/2019

Deep Learning for Predicting Dynamic Uncertain Opinions in Network Data

Subjective Logic (SL) is one of well-known belief models that can explic...

Please sign up or login with your details

Forgot password? Click here to reset