LinearModelValidation¶

(Source code, png)

class LinearModelValidation(*args)¶

Validate a linear regression metamodel.

Warning

This class is experimental and likely to be modified in future releases. To use it, import the openturns.experimental submodule.

Parameters:

resultLinearModelResult: A linear model result resulting from linear least squares regression.
splitterSplitterImplementation: The cross-validation method. The default is LeaveOneOutSplitter.

Methods

`computeMeanSquaredError`()	Accessor to the mean squared error.
`computeR2Score`()	Compute the R2 score.
`drawValidation`()	Plot a model vs metamodel graph for visual validation.
`getClassName`()	Accessor to the object's name.
`getLinearModelResult`()	Get the linear model result.
`getMetamodelPredictions`()	Accessor to the output predictions from the metamodel.
`getName`()	Accessor to the object's name.
`getOutputSample`()	Accessor to the output sample.
`getResidualDistribution`([smooth])	Compute the non parametric distribution of the residual sample.
`getResidualSample`()	Compute the residual sample.
`getSplitter`()	Get the cross-validation method.
`hasName`()	Test if the object is named.
`setName`(name)	Accessor to the object's name.

See also

openturns.LinearModelResult

Notes

A LinearModelValidation object is used for the validation of a linear model. It is based on the fast (analytical) leave-one-out and fast K-Fold cross-validation methods presented in Validation and cross validation of metamodels.

Analytical cross-validation can only be performed if all coefficients are estimated without model selection: if the coefficients are computed with model selection, then an exception is produced by default. This is because model selection leads to supposedly improved coefficients, so that the hypotheses required to estimate the mean squared error using the cross-validation method are not satisfied anymore. As a consequence, using the analytical formula without taking into account for the model selection leads to a biased, optimistic, mean squared error. More precisely, the analytical formula produces a MSE which is lower than the true one on average. Model selection is involved if the LinearModelStepwiseAlgorithm class is involved. If the LinearModelAlgorithm class is used, then no model selection is involved and the analytical cross-validation methods can be used. If model selection is involved, the naive methods based on the LeaveOneOutSplitter and KFoldSplitter classes can be used directly, but this can be much slower than the analytical methods implemented in the LinearModelValidation class. In many cases, however, the order of magnitude of the estimate from the analytical formula applied to a sparse model is correct: the estimate of the MSE is only slightly lower than the true value. In order to enable the calculation of the analytical MSE estimator on a sparse model, please set the LinearModelValidation-ModelSelection key of the ResourceMap to False: use this option at your own risks.

We suggest to use leave-one-out cross validation when possible, because it produces a more accurate estimate of the error than K-Fold does. If K-Fold is required, we suggest to use the largest possible value of $k$ .

The predictions of the leave-one-one or K-Fold surrogate models are based on the hold-out method. For example, if we use the leave-one-out cross-validation method, the $i$ -th prediction is the prediction of the linear model trained using the hold-out sample where the $i$ -th observation was removed. This produces a sample of residuals which can be retrieved using the getResidualSample method. The drawValidation performs similarly.

Examples

Create a linear model.

>>> import openturns as ot
>>> import openturns.experimental as otexp
>>> ot.RandomGenerator.SetSeed(0)
>>> func = ot.SymbolicFunction(
...     ['x1', 'x2', 'x3'],
...     ['x1 + x2 + sin(x2 * 2 * pi_) / 5 + 1e-3 * x3^2']
... )
>>> dimension = 3
>>> distribution = ot.JointDistribution([ot.Normal()] * dimension)
>>> sampleSize = 20
>>> inputSample = distribution.getSample(sampleSize)
>>> outputSample = func(inputSample)
>>> algo = ot.LinearModelAlgorithm(inputSample, outputSample)
>>> algo.run()
>>> result = algo.getResult()

Validate the linear model using leave-one-out cross-validation.

>>> validation = otexp.LinearModelValidation(result)

We can use a specific cross-validation splitter if needed.

>>> splitterLOO = ot.LeaveOneOutSplitter(sampleSize)
>>> validation = otexp.LinearModelValidation(result, splitterLOO)
>>> r2Score = validation.computeR2Score()
>>> print('R2 = ', r2Score[0])
R2 =  0.98...

Validate the linear model using K-Fold cross-validation.

>>> splitterKFold = ot.KFoldSplitter(sampleSize)
>>> validation = otexp.LinearModelValidation(result, splitterKFold)

Validate the linear model using K-Fold cross-validation and set K.

>>> kFoldParameter = 10
>>> splitterKFold = ot.KFoldSplitter(sampleSize, kFoldParameter)
>>> validation = otexp.LinearModelValidation(result, splitterKFold)

Draw the validation graph.

>>> graph = validation.drawValidation()

__init__(*args)¶

computeMeanSquaredError()¶

Accessor to the mean squared error.

Returns:

meanSquaredErrorPoint: The mean squared error of each marginal output dimension.

Notes

The sample mean squared error is:

$\widehat{\operatorname{MSE}} = \frac{1}{n} \sum_{j=1}^{n} \left(y^{(j)} - \tilde{g}\left(\bdx^{(j)}\right)\right)^2$

where $n \in \Nset$ is the sample size, $\tilde{g}$ is the metamodel, $\{\bdx^{(j)} \in \Rset^{n_X}\}_{j = 1, ..., n}$ is the input experimental design and $\{y^{(j)} \in \Rset\}_{j = 1, ..., n}$ is the output of the model.

If the output is multi-dimensional, the same calculations are repeated separately for each output marginal $k$ for $k = 1, ..., n_y$ where $n_y \in \Nset$ is the output dimension.

computeR2Score()¶

Compute the R2 score.

Returns:

r2ScorePoint: The coefficient of determination R2

Notes

The coefficient of determination $R^2$ is the fraction of the variance of the output explained by the metamodel. It is defined as:

$R^2 = 1 - \operatorname{FVU}$

where $\operatorname{FVU}$ is the fraction of unexplained variance:

$\operatorname{FVU} = \frac{\operatorname{MSE}(\tilde{g}) }{\Var{Y}}$

where $Y = g(\bdX)$ is the output of the physical model $g$ , $\Var{Y}$ is the variance of the output and $\operatorname{MSE}$ is the mean squared error of the metamodel:

$\operatorname{MSE}(\tilde{g}) = \Expect{\left(g(\bdX) - \tilde{g}(\bdX) \right)^2}.$

The sample $R^2$ is:

$\hat{R}^2 = 1 - \frac{\frac{1}{n} \sum_{j=1}^{n} \left(y^{(j)} - \tilde{g}\left(\bdx^{(j)}\right)\right)^2}{\hat{\sigma}^2_Y}$

where $n \in \Nset$ is the sample size, $\tilde{g}$ is the metamodel, $\left\{\bdx^{(j)} \in \Rset^{n_X}\right\}_{j = 1, ..., n}$ is the input experimental design, $\left\{y^{(j)} \in \Rset\right\}_{j = 1, ..., n}$ is the output of the model and $\hat{\sigma}^2_Y$ is the sample variance of the output:

$\hat{\sigma}^2_Y = \frac{1}{n - 1} \sum_{j=1}^{n} \left(y^{(j)} - \overline{y}\right)^2$

where $\overline{y}$ is the output sample mean:

$\overline{y} = \frac{1}{n} \sum_{j=1}^{n} y^{(j)}.$

drawValidation()¶

Plot a model vs metamodel graph for visual validation.

Returns:

graphGridLayout: The visual validation graph.

Notes

The plot presents the metamodel predictions depending on the model observations. If the points are close to the diagonal line of the plot, then the metamodel validation is satisfactory. Points which are far away from the diagonal represent outputs for which the metamodel is not accurate.

If the output is multi-dimensional, the graph has 1 row and $n_y \in \Nset$ columns, where $n_y$ is the output dimension.

getClassName()¶

Accessor to the object’s name.

Returns:

class_namestr: The object class name (object.__class__.__name__).

getLinearModelResult()¶

Get the linear model result.

Returns:

linearModelResultLinearModelResult: The linear model result.

getMetamodelPredictions()¶

Accessor to the output predictions from the metamodel.

Returns:

outputMetamodelSampleSample: Output sample of the metamodel.

getName()¶

Accessor to the object’s name.

Returns:

namestr: The name of the object.

getOutputSample()¶

Accessor to the output sample.

Returns:

outputSampleSample: Output sample of a model evaluated apart.

getResidualDistribution(smooth=True)¶

Compute the non parametric distribution of the residual sample.

Parameters:

smoothbool: Tells if distribution is smooth (true) or not. Default argument is true.

Returns:

residualDistributionDistribution: The residual distribution.

Notes

The residual distribution is built thanks to KernelSmoothing if smooth argument is true. Otherwise, an histogram distribution is returned, thanks to HistogramFactory.

getResidualSample()¶

Compute the residual sample.

Returns:

residualSample: The residual sample.

Notes

The residual sample is given by :

$r^{(j)} = y^{(j)} - \tilde{g}\left(\vect{x}^{(j)}\right)$

for $j = 1, ..., n$ where $n \in \Nset$ is the sample size, $y^{(j)}$ is the model observation, $\tilde{g}$ is the metamodel and $\vect{x}^{(j)}$ is the $j$ -th input observation.

If the output is multi-dimensional, the residual sample has dimension $n_y \in \Nset$ , where $n_y$ is the output dimension.

getSplitter()¶

Get the cross-validation method.

Returns:

splitterSplitterImplementation: The cross-validation method.

hasName()¶

Test if the object is named.

Returns:

hasNamebool: True if the name is not empty.

setName(name)¶

Accessor to the object’s name.

Parameters:

namestr: The name of the object.

OpenTURNS

An Open source initiative for the Treatment of Uncertainties, Risks'N Statistics

Previous topic

Next topic

This Page

LinearModelValidation¶