Two-sample Kolmogorov-Smirnov testΒΆ

The Kolmogorov-Smirnov test is a statistical test that can be used to test whether two given samples of data are drawn from the same distribution which is of dimension 1 and continuous.

Let \left\{ x_1,\ldots,x_{\sampleSize} \right\} and \left\{ y_1,\ldots,y_{\sampleSize} \right\} be two samples of dimension 1 respectively drawn from the (unknown) distribution functions F_X and F_Y.

We want to test whether both samples are drawn from the same distribution, ie whether F_X=F_Y.

This test involves the calculation of the test statistic which is the weighted maximum distance between both empirical cumulative distribution function F_{\sampleSize, X} and F_{\sampleSize, Y}. Letting X_1, \ldots , X_{\sampleSize} and Y_1, \ldots , Y_{\sampleSize} be independent random variables respectively distributed according to X and Y, both empirical cumulative distribution functions are defined by:

F_{\sampleSize, X}(x) & = \sum_{i=1}^{\sampleSize} 1_{X_i \leq x} \\ F_{\sampleSize, Y}(x) & = \sum_{i=1}^{\sampleSize} 1_{Y_i \leq x}

for all :math:` x in Rset`. The test statistic is defined by:

D_{\sampleSize} = \sqrt{\sampleSize} \sup_x \left|F_{\sampleSize, X}\left(x\right) - F_{\sampleSize, Y}\left(x\right)\right|

The empirical value of the test statistic is denoted by d, using the realization of F_{\sampleSize, X} and F_{\sampleSize, Y} on the samples:

F_{\sampleSize, X}(x) & = \dfrac{\mbox{number of } x_i \leq x \mbox{ in the sample}}{\sampleSize}\\ F_{\sampleSize, Y}(x) & = \dfrac{\mbox{number of } x_i \leq x \mbox{ in the sample}}{\sampleSize}

Under the null hypothesis \mathcal{H}_0 = \{ F_X = F_Y\}, the distribution of the test statistic D_{\sampleSize} is known: algorithms are available to compute the distribution of \widehat{D} both for \sampleSize large (asymptotic distribution: this is the Kolmogorov distribution) or for \sampleSize small (exact distribution). Then we can use that distribution to apply the test as follows. We fix a risk \alpha (error type I) and we evaluate the associated critical value d_\alpha which is the quantile of order 1-\alpha of D_{\sampleSize}.

Then a decision is made, either by comparing the test statistic to the theoretical threshold d_\alpha (or equivalently by evaluating the p-value of the sample defined as \Prob{D_{\sampleSize} > d_{\sampleSize}} and by comparing it to \alpha):

  • if d_{\sampleSize}>d_{\alpha} (or equivalently \Prob{D_{\sampleSize} > d_{\sampleSize}} < \alpha), then we reject the null hypothesis according to which both samples are drawn from the same distribution,

  • if d_{\sampleSize} \leq d_{\alpha} (or equivalently \Prob{D_{\sampleSize} > d_{\sampleSize}} \geq \alpha), then the null hypothesis is considered acceptable.