Dataset balancing can hurt model performance

by   R. Channing Moore, et al.

Machine learning from training data with a skewed distribution of examples per class can lead to models that favor performance on common classes at the expense of performance on rare ones. AudioSet has a very wide range of priors over its 527 sound event classes. Classification performance on AudioSet is usually evaluated by a simple average over per-class metrics, meaning that performance on rare classes is equal in importance to the performance on common ones. Several recent papers have used dataset balancing techniques to improve performance on AudioSet. We find, however, that while balancing improves performance on the public AudioSet evaluation data it simultaneously hurts performance on an unpublished evaluation set collected under the same conditions. By varying the degree of balancing, we show that its benefits are fragile and depend on the evaluation set. We also do not find evidence indicating that balancing improves rare class performance relative to common classes. We therefore caution against blind application of balancing, as well as against paying too much attention to small improvements on a public evaluation set.


page 1

page 2

page 3

page 4


An Exploration of How Training Set Composition Bias in Machine Learning Affects Identifying Rare Objects

When training a machine learning classifier on data where one of the cla...

Using Multitask Learning to Improve 12-Lead Electrocardiogram Classification

We develop a multi-task convolutional neural network (CNN) to classify m...

Revisiting Long-tailed Image Classification: Survey and Benchmarks with New Evaluation Metrics

Recently, long-tailed image classification harvests lots of research att...

The Effect of Balancing Methods on Model Behavior in Imbalanced Classification Problems

Imbalanced data poses a significant challenge in classification as model...

A Novel Resampling Technique for Imbalanced Dataset Optimization

Despite the enormous amount of data, particular events of interest can s...

FOCUS: Familiar Objects in Common and Uncommon Settings

Standard training datasets for deep learning often contain objects in co...

Please sign up or login with your details

Forgot password? Click here to reset