Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowds

by   Shikun Li, et al.

Learning from crowds describes that the annotations of training data are obtained with crowd-sourcing services. Multiple annotators each complete their own small part of the annotations, where labeling mistakes that depend on annotators occur frequently. Modeling the label-noise generation process by the noise transition matrix is a power tool to tackle the label noise. In real-world crowd-sourcing scenarios, noise transition matrices are both annotator- and instance-dependent. However, due to the high complexity of annotator- and instance-dependent transition matrices (AIDTM), annotation sparsity, which means each annotator only labels a little part of instances, makes modeling AIDTM very challenging. Prior works simplify the problem by assuming the transition matrix is instance-independent or using simple parametric way, while lose modeling generality. Motivated by this, we target a more realistic problem, estimating general AIDTM in practice. Without losing modeling generality, we parameterize AIDTM with deep neural networks. To alleviate the modeling challenge, we suppose every annotator shares its noise pattern with similar annotators, and estimate AIDTM via knowledge transfer. We hence first model the mixture of noise patterns by all annotators, and then transfer this modeling to individual annotators. Furthermore, considering that the transfer from the mixture of noise patterns to individuals may cause two annotators with highly different noise generations to perturb each other, we employ the knowledge transfer between identified neighboring annotators to calibrate the modeling. Experiments confirm the superiority of the proposed approach on synthetic and real-world crowd-sourcing data. Source codes will be released.


page 1

page 2

page 3

page 4


Parts-dependent Label Noise: Towards Instance-dependent Label Noise

Learning with the instance-dependent label noise is challenging, because...

Rethinking the Value of Labels for Instance-Dependent Label Noise Learning

Label noise widely exists in large-scale datasets and significantly dege...

Instance-dependent Label-noise Learning under a Structural Causal Model

Label noise will degenerate the performance of deep learning algorithms ...

Instance-Dependent Label-Noise Learning with Manifold-Regularized Transition Matrix Estimation

In label-noise learning, estimating the transition matrix has attracted ...

Learning from Crowds by Modeling Common Confusions

Crowdsourcing provides a practical way to obtain large amounts of labele...

Approximating Instance-Dependent Noise via Instance-Confidence Embedding

Label noise in multiclass classification is a major obstacle to the depl...

Please sign up or login with your details

Forgot password? Click here to reset