Learning to Benchmark: Determining Best Achievable Misclassification Error from Training Data

by   Morteza Noshad, et al.

We address the problem of learning to benchmark the best achievable classifier performance. In this problem the objective is to establish statistically consistent estimates of the Bayes misclassification error rate without having to learn a Bayes-optimal classifier. Our learning to benchmark framework improves on previous work on learning bounds on Bayes misclassification rate since it learns the exact Bayes error rate instead of a bound on error rate. We propose a benchmark learner based on an ensemble of ϵ-ball estimators and Chebyshev approximation. Under a smoothness assumption on the class densities we show that our estimator achieves an optimal (parametric) mean squared error (MSE) rate of O(N^-1), where N is the number of samples. Experiments on both simulated and real datasets establish that our proposed benchmark learning algorithm produces estimates of the Bayes error that are more accurate than previous approaches for learning bounds on Bayes error probability.


page 1

page 2

page 3

page 4


Rate-optimal Meta Learning of Classification Error

Meta learning of optimal classifier error rates allows an experimenter t...

Evaluating State-of-the-Art Classification Models Against Bayes Optimality

Evaluating the inherent difficulty of a given data-driven classification...

Evaluating Bayes Error Estimators on Read-World Datasets with FeeBee

The Bayes error rate (BER) is a fundamental concept in machine learning ...

Fast Meta-Learning for Adaptive Hierarchical Classifier Design

We propose a new splitting criterion for a meta-learning approach to mul...

Learning to Bound the Multi-class Bayes Error

In the context of supervised learning, meta learning uses features, meta...

Equivalence between Time Series Predictability and Bayes Error Rate

Predictability is an emerging metric that quantifies the highest possibl...

Please sign up or login with your details

Forgot password? Click here to reset