Compute reference probabilities with Monte-Carlo

In this example, we perform a reliability benchmark based on a large Monte-Carlo sample. In order to limit the elapsed time, we consider a limited elapsed time for each problem. In order to get the best possible accuracy within this time limit, we set the coefficient of variation to zero.

import otbenchmark as otb
import pandas as pd
import numpy as np
from tqdm import tqdm
import time
problemslist = otb.ReliabilityBenchmarkProblemList()
numberOfProblems = len(problemslist)
numberOfProblems
26
coefficientOfVariation = 0.0
maximumOuterSampling = 10 ** 4  # 10 ** 6 for real
blockSize = 10 ** 0  # 10 ** 4 for real simulations
blockSize
1
confidenceLevel = 0.95
maximumDurationSeconds = 5 * 60.0
totalDurationMinutes = numberOfProblems * maximumDurationSeconds / 60.0
totalDurationMinutes
130.0
model_names = [problemslist[i].getName() for i in range(numberOfProblems)]
metrics = ["PF", "N. function calls", "PMin", "PMax", "C.O.V.", "Digits", "Time (s)"]
resultArray = np.zeros((numberOfProblems, len(metrics)))
for i in tqdm(range(numberOfProblems)):
    startTime = time.time()
    problem = problemslist[i]
    name = problem.getName()
    event = problem.getEvent()
    g = event.getFunction()
    factory = otb.ProbabilitySimulationAlgorithmFactory()
    algo = factory.buildMonteCarlo(problem)
    algo.setMaximumOuterSampling(maximumOuterSampling)
    algo.setBlockSize(blockSize)
    algo.setMaximumCoefficientOfVariation(coefficientOfVariation)
    algo.setMaximumTimeDuration(maximumDurationSeconds)
    initialNumberOfCall = g.getEvaluationCallsNumber()
    algo.run()
    result = algo.getResult()
    numberOfFunctionEvaluations = g.getEvaluationCallsNumber() - initialNumberOfCall
    computedProbability = result.getProbabilityEstimate()
    confidenceLength = result.getConfidenceLength(confidenceLevel)
    pmin = computedProbability - 0.5 * confidenceLength
    pmax = computedProbability + 0.5 * confidenceLength
    cov = result.getCoefficientOfVariation()
    if cov > 0.0:
        expectedDigits = -np.log10(cov) - 1.0
    else:
        expectedDigits = 0.0
    stopTime = time.time()
    elapsedTime = stopTime - startTime
    resultArray[i][0] = computedProbability
    resultArray[i][1] = numberOfFunctionEvaluations
    resultArray[i][2] = pmin
    resultArray[i][3] = pmax
    resultArray[i][4] = cov
    resultArray[i][5] = expectedDigits
    resultArray[i][6] = elapsedTime
  0%|          | 0/26 [00:00<?, ?it/s]
 23%|██▎       | 6/26 [00:00<00:00, 58.81it/s]
 46%|████▌     | 12/26 [00:00<00:00, 59.43it/s]
 69%|██████▉   | 18/26 [00:00<00:00, 51.62it/s]
 92%|█████████▏| 24/26 [00:00<00:00, 30.45it/s]
100%|██████████| 26/26 [00:00<00:00, 37.90it/s]
df = pd.DataFrame(resultArray, index=model_names, columns=metrics)
# df.to_csv("reliability_compute_reference_proba.csv")
df
PF N. function calls PMin PMax C.O.V. Digits Time (s)
RP8 0.0006 10000.0 0.000120 0.001080 0.408126 -0.610794 0.021044
RP14 0.0010 10000.0 0.000381 0.001619 0.316070 -0.499783 0.020689
RP22 0.0045 10000.0 0.003188 0.005812 0.148735 -0.172414 0.014276
RP24 0.0030 10000.0 0.001928 0.004072 0.182300 -0.260787 0.015093
RP25 0.0000 10000.0 0.000000 0.000000 -1.000000 0.000000 0.015648
RP28 0.0000 10000.0 0.000000 0.000000 -1.000000 0.000000 0.015205
RP31 0.0024 10000.0 0.001441 0.003359 0.203879 -0.309373 0.014385
RP33 0.0024 10000.0 0.001441 0.003359 0.203879 -0.309373 0.016393
RP35 0.0022 10000.0 0.001282 0.003118 0.212966 -0.328310 0.014730
RP38 0.0101 10000.0 0.008140 0.012060 0.099000 0.004365 0.024060
RP53 0.0328 10000.0 0.029309 0.036291 0.054303 0.265179 0.014597
RP55 0.5627 10000.0 0.552978 0.572422 0.008816 1.054749 0.015823
RP54 0.0006 10000.0 0.000120 0.001080 0.408126 -0.610794 0.042482
RP57 0.0308 10000.0 0.027414 0.034186 0.056096 0.251069 0.015019
RP75 0.0100 10000.0 0.008050 0.011950 0.099499 0.002182 0.014320
RP89 0.0054 10000.0 0.003964 0.006836 0.135715 -0.132627 0.014773
RP107 0.0000 10000.0 0.000000 0.000000 -1.000000 0.000000 0.031306
RP110 0.0000 10000.0 0.000000 0.000000 -1.000000 0.000000 0.016305
RP111 0.0000 10000.0 0.000000 0.000000 -1.000000 0.000000 0.018281
RP63 0.0003 10000.0 -0.000039 0.000639 0.577264 -0.761374 0.231111
RP91 0.0008 10000.0 0.000246 0.001354 0.353412 -0.548281 0.020761
RP60 0.0441 10000.0 0.040076 0.048124 0.046557 0.332013 0.018593
RP77 0.0000 10000.0 0.000000 0.000000 -1.000000 0.000000 0.017123
Four-branch serial system 0.0035 10000.0 0.002343 0.004657 0.168735 -0.227205 0.014605
R-S 0.0755 10000.0 0.070322 0.080678 0.034993 0.456020 0.014605
Axial stressed beam 0.0292 10000.0 0.025900 0.032500 0.057660 0.239127 0.013594


The problems with higher failture probabilities are obviously solved with more accuracy with the Monte-Carlo method. For example, the RP55 problem which has the highest probability equal to 0.560 has more than 3 significant digits. On the opposite side, the problems with probabilities close to zero are much more difficult to solve. The RP28 with a probability close to 10^{-7} has no significant digit.

These previous results are consistent with the distribution of the Monte-Carlo estimator. The properties of the binomial distribution imply that its variance is:

\sigma_{p_f}^2 = \frac{p_f (1-p_f)}{n}

where n is the sample size and p_f is the failure probability. The coefficient of variation is:

CV = \frac{\sigma_{p_f}}{p_f}.

Since we do not know the exact value of p_f, we use is approximation \tilde{p_f} instead (this turns rigorous equations into approximate ones, but does not change the outcome). This implies:

CV = \sqrt{\frac{1 - p_f}{p_f n}}.

When p_f\rightarrow 0, we have p_f \rightarrow 0 which implies:

CV \rightarrow \sqrt{\frac{1}{p_f n}}.

Inverting the previous equation, we get the sample size given the coefficient of variation:

n \approx \frac{1}{p_f CV^2}.

This leads to the rule of thumb that, in order to estimate the probability p_f = 10^{-m}, where m is an integer, we need a sample size equal to:

n \approx \frac{1}{10^{-m} 10^{-2}} = 10^{m+2}.

For example, estimating the probability of the RP28 problem with just one single digit leads to a sample size equal to n=10^9, since the exact p_f \approx 10^{-7}.

Total running time of the script: (0 minutes 0.874 seconds)