Regression analysis¶
In this page, we analyse the result from linear regression. These statistics make it possible to analyse a Linear regression.
The linear model is:
where:
is the dimension of the input vector,
is the input vector,
is the number of parameters,
are the basis functions where for ,
are the coefficients where for ,
where is a normal distribution and is its standard deviation.
The main goal of considering a normal noise is to be able to make tests of significance (see [rawlings2001] page 3). In particular, this enables to use the F-test and T-test that we are going to review later in this document. Furthermore, if the errors are normal, then the method of least squares and the maximum likelihood method are equivalent (see [bingham2010] theorem 1.8 page 22).
Caution¶
There is an ambiguity when the number of parameters is unspecified in the text, which may explain the differences between the various formulas we find in the books. In some texts, the intercept has the index 0, which leads to an increased number of parameters. In the present document, the number of parameters is equal to , but not all books use the same convention:
in [baron2014] (eq. 11.11 page 396), in [sen1990] (eq. 2.1 page 28) and in [rawlings2001] (eq. 3.1 page 75), the number of parameters is equal to ,
in [bingham2010] (page 61) and in [faraway2014] (page 15), the number of parameters is equal to .
Experimental design¶
Let be the sample size. A set of observations of the input variables is required:
We assume that the random errors are independent. We consider the corresponding model evaluations:
where:
for . Since the errors are independent, then the output observations are independent too. Let be the vector of output observations.
Solution of the least squares problem¶
The design matrix is the value of the basis functions over the inputs variables in the sample:
for and . Assume that the design matrix has full rank. The solution of the linear least squares problem is:
Statistics¶
Let be the sample mean:
The total sum of squares (see [baron2014] page 398) is:
where . The regression sum of squares is:
The error sum of squares is:
Coefficient of determination¶
The coefficient of determination is (see [baron2014] page 399):
The coefficient of determination measures the part of the variance explained by the linear regression model.
Variance¶
The unbiased estimator of the variance is (see [baron2014] page 400, [bingham2010] page 67):
ANOVA F-test¶
The F-statistic is based on the hypothesis that all coefficients are simultaneously zero (see [baron2014] page 400). More precisely, the ANOVA F-test is based on the hypothesis:
Let be the vector of observations and be the vector of predictions.
The F-statistic is (see [bingham2010] Kolodziejcyzk’s theorem 6.5 page 154, [baron2014] page 400):
The p-value is computed from the Fisher-Snedecor distribution (see [baron2014] page 400, [faraway2014] page 35).
T-test for individual coefficients¶
The T-test is based on the hypothesis that one single coefficient is zero. More precisely, let be the estimator of the variance (see [baron2014] page 400):
The variance of the estimator of the parameters is:
Let be the standard deviation of the estimator of :
for any . For , the T-test is (see [baron2014] page 401):
The T-statistic is (see [baron2014] page 401, [rawlings2001] eq. 4.47 page 122):
for . The p-value is computed from the Student’s T distribution with degrees of freedom (see [baron2014] page 401).