Theory of Curriculum Learning, with Convex Loss Functions

by   Daphna Weinshall, et al.

Curriculum Learning - the idea of teaching by gradually exposing the learner to examples in a meaningful order, from easy to hard, has been investigated in the context of machine learning long ago. Although methods based on this concept have been empirically shown to improve performance of several learning algorithms, no theoretical analysis has been provided even for simple cases. To address this shortfall, we start by formulating an ideal definition of difficulty score - the loss of the optimal hypothesis at a given datapoint. We analyze the possible contribution of curriculum learning based on this score in two convex problems - linear regression, and binary classification by hinge loss minimization. We show that in both cases, the expected convergence rate decreases monotonically with the ideal difficulty score, in accordance with earlier empirical results. We also prove that when the ideal difficulty score is fixed, the convergence rate is monotonically increasing with respect to the loss of the current hypothesis at each point. We discuss how these results bring to term two apparently contradicting heuristics: curriculum learning on the one hand, and hard data mining on the other.


page 1

page 2

page 3

page 4


Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks

Our first contribution in this paper is a theoretical investigation of c...

Spatial Transformer Networks for Curriculum Learning

Curriculum learning is a bio-inspired training technique that is widely ...

Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification

Applying curriculum learning requires both a range of difficulty in data...

Target Curricula via Selection of Minimum Feature Sets: a Case Study in Boolean Networks

We consider the effect of introducing a curriculum of targets when train...

Adversarial Focal Loss: Asking Your Discriminator for Hard Examples

Focal Loss has reached incredible popularity as it uses a simple techniq...

Understanding Self-Paced Learning under Concave Conjugacy Theory

By simulating the easy-to-hard learning manners of humans/animals, the l...

An Analytical Theory of Curriculum Learning in Teacher-Student Networks

In humans and animals, curriculum learning – presenting data in a curate...

Please sign up or login with your details

Forgot password? Click here to reset