Hopkins statistic

The Hopkins statistic (introduced by Brian Hopkins and John Gordon Skellam) is a way of measuring the cluster tendency of a data set.^[1] It belongs to the family of sparse sampling tests. It acts as a statistical hypothesis test where the null hypothesis is that the data is generated by a Poisson point process and are thus uniformly randomly distributed.^[2] A value close to 1 tends to indicate the data is highly clustered, random data will tend to result in values around 0.5, and uniformly distributed data will tend to result in values close to 0.^[3]

Preliminaries[]

A typical formulation of the Hopkins statistic follows.^[2]

Let ${\displaystyle X}$ be the set of ${\displaystyle n}$ data points.

Consider a random sample (without replacement) of ${\displaystyle m\ll n}$ data points with members ${\displaystyle x_{i}}$.

Generate a set ${\displaystyle Y}$ of ${\displaystyle m}$ uniformly randomly distributed data points.

Define two distance measures,

${\displaystyle u_{i},}$ the distance of ${\displaystyle y_{i}\in Y}$ from its nearest neighbour in ${\displaystyle X}$, and

${\displaystyle w_{i},}$ the distance of ${\displaystyle m}$ number of randomly chosen ${\displaystyle x_{i},}$ ${\displaystyle x_{i}\in X}$ from its nearest neighbour in ${\displaystyle X}$.

Definition[]

With the above notation, if the data is ${\displaystyle d}$ dimensional, then the Hopkins statistic is defined as:^[4]

${\displaystyle H={\frac {\sum _{i=1}^{m}{u_{i}^{d}}}{\sum _{i=1}^{m}{u_{i}^{d}}+\sum _{i=1}^{m}{w_{i}^{d}}}}\,}$

Under the null hypotheses, this statistic has a Beta(m,m) distribution.

Notes and references[]

External links[]

• http://www.sthda.com/english/wiki/assessing-clustering-tendency-a-vital-issue-unsupervised-machine-learning