Ground Truth Bias in External Cluster Validity Indices

by   Yang Lei, et al.

It has been noticed that some external CVIs exhibit a preferential bias towards a larger or smaller number of clusters which is monotonic (directly or inversely) in the number of clusters in candidate partitions. This type of bias is caused by the functional form of the CVI model. For example, the popular Rand index (RI) exhibits a monotone increasing (NCinc) bias, while the Jaccard Index (JI) index suffers from a monotone decreasing (NCdec) bias. This type of bias has been previously recognized in the literature. In this work, we identify a new type of bias arising from the distribution of the ground truth (reference) partition against which candidate partitions are compared. We call this new type of bias ground truth (GT) bias. This type of bias occurs if a change in the reference partition causes a change in the bias status (e.g., NCinc, NCdec) of a CVI. For example, NCinc bias in the RI can be changed to NCdec bias by skewing the distribution of clusters in the ground truth partition. It is important for users to be aware of this new type of biased behaviour, since it may affect the interpretations of CVI results. The objective of this article is to study the empirical and theoretical implications of GT bias. To the best of our knowledge, this is the first extensive study of such a property for external cluster validity indices.


page 1

page 2

page 3

page 4


Adjusted Asymmetric Accuracy: A Well-Behaving External Cluster Validity Measure

There is no, nor will there ever be, single best clustering algorithm, b...

Comprehensive cluster validity Index based on structural simplicity

Nonhierarchical clustering depending on unsupervised algorithms may not ...

Identifying bias in cluster quality metrics

We study potential biases of popular cluster quality metrics, such as co...

A new approach for evaluating internal cluster validation indices

A vast number of different methods are available for unsupervised classi...

Are Cluster Validity Measures (In)valid?

Internal cluster validity measures (such as the Calinski-Harabasz, Dunn,...

Measuring Player's Behaviour Change over Time in Public Goods Game

An important issue in public goods game is whether player's behaviour ch...

Comparing Two Partitions of Non-Equal Sets of Units

Rand (1971) proposed what has since become a well-known index for compar...

Please sign up or login with your details

Forgot password? Click here to reset