Multi-Dimensional Ability Diagnosis for Machine Learning Algorithms

by   Qi Liu, et al.

Machine learning algorithms have become ubiquitous in a number of applications (e.g. image classification). However, due to the insufficient measurement of traditional metrics (e.g. the coarse-grained Accuracy of each classifier), substantial gaps are usually observed between the real-world performance of these algorithms and their scores in standardized evaluations. In this paper, inspired by the psychometric theories from human measurement, we propose a task-agnostic evaluation framework Camilla, where a multi-dimensional diagnostic metric Ability is defined for collaboratively measuring the multifaceted strength of each machine learning algorithm. Specifically, given the response logs from different algorithms to data samples, we leverage cognitive diagnosis assumptions and neural networks to learn the complex interactions among algorithms, samples and the skills (explicitly or implicitly pre-defined) of each sample. In this way, both the abilities of each algorithm on multiple skills and some of the sample factors (e.g. sample difficulty) can be simultaneously quantified. We conduct extensive experiments with hundreds of machine learning algorithms on four public datasets, and our experimental results demonstrate that Camilla not only can capture the pros and cons of each algorithm more precisely, but also outperforms state-of-the-art baselines on the metric reliability, rank consistency and rank stability.


Solving machine learning optimization problems using quantum computers

Classical optimization algorithms in machine learning often take a long ...

Towards Better Fairness-Utility Trade-off: A Comprehensive Measurement-Based Reinforcement Learning Framework

Machine learning is widely used to make decisions with societal impact s...

Not quite unreasonable effectiveness of machine learning algorithms

State-of-the-art machine learning algorithms demonstrate close to absolu...

Identifiable Cognitive Diagnosis with Encoder-decoder for Modelling Students' Performance

Cognitive diagnosis aims to diagnose students' knowledge proficiencies b...

Relevance As a Metric for Evaluating Machine Learning Algorithms

In machine learning, the choice of a learning algorithm that is suitable...

Mitigating Spreadsheet Risk in Complex Multi-Dimensional Models in Excel

Microsoft Excel is the most ubiquitous analytical tool ever built. Compa...

Please sign up or login with your details

Forgot password? Click here to reset