.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_check_reliability_reference_probabilities.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_plot_check_reliability_reference_probabilities.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_check_reliability_reference_probabilities.py:


Check reliability problems reference probabilities
==================================================

.. GENERATED FROM PYTHON SOURCE LINES 7-25

In this example, we check that the reference probabilities in the reliability problems
are consistent with confidence bounds from Monte-Carlo simulations.
These 95% confidence bounds are stored in 'reliability_compute_reference_proba.csv'
and required approximately than :math:`10^9` function evaluations for each problem.

We consider two different metrics:

* we check if the reference probability is within the 95% confidence bounds,
* we compute the number of significant digits by comparing the Monte-Carlo estimator and the reference value.

The number of significant digits may be as high as 17 when all decimal digits are correct.
However, the reference probabilities are only known up to 3 digits for most problems.
In order to keep some safeguard, we will be happy with 2 correct digits.

These metrics may fail.

* On average, a fraction equal to 5% of the estimates are outside the confidence bounds.
* The Monte-Carlo estimator may not be accurate enough, e.g. if the probability is very close to zero.

.. GENERATED FROM PYTHON SOURCE LINES 27-31

.. code-block:: Python

    import otbenchmark as otb
    import pandas as pd









.. GENERATED FROM PYTHON SOURCE LINES 32-36

.. code-block:: Python

    df = pd.read_csv("reliability_compute_reference_proba.csv")
    df







.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Unnamed: 0</th>
          <th>PF</th>
          <th>N. function calls</th>
          <th>PMin</th>
          <th>PMax</th>
          <th>C.O.V.</th>
          <th>Digits</th>
          <th>Time (s)</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>RP8</td>
          <td>7.908179e-04</td>
          <td>2.413400e+08</td>
          <td>7.872713e-04</td>
          <td>7.943646e-04</td>
          <td>0.002288</td>
          <td>1.640503</td>
          <td>300.011341</td>
        </tr>
        <tr>
          <th>1</th>
          <td>RP14</td>
          <td>7.708905e-04</td>
          <td>7.441900e+08</td>
          <td>7.688964e-04</td>
          <td>7.728846e-04</td>
          <td>0.001320</td>
          <td>1.879484</td>
          <td>300.004585</td>
        </tr>
        <tr>
          <th>2</th>
          <td>RP22</td>
          <td>4.207357e-03</td>
          <td>1.495610e+09</td>
          <td>4.204076e-03</td>
          <td>4.210637e-03</td>
          <td>0.000398</td>
          <td>2.400308</td>
          <td>300.000947</td>
        </tr>
        <tr>
          <th>3</th>
          <td>RP24</td>
          <td>2.860848e-03</td>
          <td>1.644700e+09</td>
          <td>2.858266e-03</td>
          <td>2.863429e-03</td>
          <td>0.000460</td>
          <td>2.336891</td>
          <td>300.001723</td>
        </tr>
        <tr>
          <th>4</th>
          <td>RP25</td>
          <td>4.175883e-05</td>
          <td>1.567860e+09</td>
          <td>4.143895e-05</td>
          <td>4.207871e-05</td>
          <td>0.003908</td>
          <td>1.408015</td>
          <td>300.015279</td>
        </tr>
        <tr>
          <th>5</th>
          <td>RP28</td>
          <td>1.315725e-07</td>
          <td>1.839290e+09</td>
          <td>1.149947e-07</td>
          <td>1.481503e-07</td>
          <td>0.064286</td>
          <td>0.191886</td>
          <td>300.000353</td>
        </tr>
        <tr>
          <th>6</th>
          <td>RP31</td>
          <td>3.227556e-03</td>
          <td>1.787260e+09</td>
          <td>3.224926e-03</td>
          <td>3.230186e-03</td>
          <td>0.000416</td>
          <td>2.381211</td>
          <td>300.000409</td>
        </tr>
        <tr>
          <th>7</th>
          <td>RP33</td>
          <td>2.574817e-03</td>
          <td>1.430700e+09</td>
          <td>2.572190e-03</td>
          <td>2.577443e-03</td>
          <td>0.000520</td>
          <td>2.283686</td>
          <td>300.000557</td>
        </tr>
        <tr>
          <th>8</th>
          <td>RP35</td>
          <td>3.478964e-03</td>
          <td>1.415140e+09</td>
          <td>3.475896e-03</td>
          <td>3.482032e-03</td>
          <td>0.000450</td>
          <td>2.346860</td>
          <td>300.002611</td>
        </tr>
        <tr>
          <th>9</th>
          <td>RP38</td>
          <td>8.059349e-03</td>
          <td>7.765700e+08</td>
          <td>8.053061e-03</td>
          <td>8.065638e-03</td>
          <td>0.000398</td>
          <td>2.399976</td>
          <td>300.003572</td>
        </tr>
        <tr>
          <th>10</th>
          <td>RP53</td>
          <td>3.131966e-02</td>
          <td>1.420390e+09</td>
          <td>3.131060e-02</td>
          <td>3.132872e-02</td>
          <td>0.000148</td>
          <td>2.831000</td>
          <td>300.002226</td>
        </tr>
        <tr>
          <th>11</th>
          <td>RP55</td>
          <td>5.600269e-01</td>
          <td>1.523630e+09</td>
          <td>5.600020e-01</td>
          <td>5.600519e-01</td>
          <td>0.000023</td>
          <td>3.643809</td>
          <td>300.000719</td>
        </tr>
        <tr>
          <th>12</th>
          <td>RP54</td>
          <td>9.927480e-04</td>
          <td>1.715400e+08</td>
          <td>9.880351e-04</td>
          <td>9.974610e-04</td>
          <td>0.002422</td>
          <td>1.615796</td>
          <td>300.018069</td>
        </tr>
        <tr>
          <th>13</th>
          <td>RP57</td>
          <td>2.822772e-02</td>
          <td>1.249510e+09</td>
          <td>2.821854e-02</td>
          <td>2.823691e-02</td>
          <td>0.000166</td>
          <td>2.779904</td>
          <td>300.000954</td>
        </tr>
        <tr>
          <th>14</th>
          <td>RP75</td>
          <td>9.818417e-03</td>
          <td>1.597510e+09</td>
          <td>9.813582e-03</td>
          <td>9.823253e-03</td>
          <td>0.000251</td>
          <td>2.599863</td>
          <td>300.000640</td>
        </tr>
        <tr>
          <th>15</th>
          <td>RP89</td>
          <td>5.469847e-03</td>
          <td>1.366180e+09</td>
          <td>5.465935e-03</td>
          <td>5.473758e-03</td>
          <td>0.000365</td>
          <td>2.437911</td>
          <td>300.000577</td>
        </tr>
        <tr>
          <th>16</th>
          <td>RP107</td>
          <td>2.754926e-07</td>
          <td>5.227000e+08</td>
          <td>2.304941e-07</td>
          <td>3.204912e-07</td>
          <td>0.083337</td>
          <td>0.079160</td>
          <td>300.004477</td>
        </tr>
        <tr>
          <th>17</th>
          <td>RP110</td>
          <td>3.183607e-05</td>
          <td>1.344010e+09</td>
          <td>3.153441e-05</td>
          <td>3.213774e-05</td>
          <td>0.004835</td>
          <td>1.315646</td>
          <td>300.001033</td>
        </tr>
        <tr>
          <th>18</th>
          <td>RP111</td>
          <td>7.851043e-07</td>
          <td>1.552660e+09</td>
          <td>7.410290e-07</td>
          <td>8.291796e-07</td>
          <td>0.028643</td>
          <td>0.542980</td>
          <td>300.000674</td>
        </tr>
        <tr>
          <th>19</th>
          <td>RP63</td>
          <td>3.772015e-04</td>
          <td>5.360000e+07</td>
          <td>3.720028e-04</td>
          <td>3.824002e-04</td>
          <td>0.007032</td>
          <td>1.152929</td>
          <td>300.044605</td>
        </tr>
        <tr>
          <th>20</th>
          <td>RP91</td>
          <td>6.998219e-04</td>
          <td>8.196500e+08</td>
          <td>6.980114e-04</td>
          <td>7.016324e-04</td>
          <td>0.001320</td>
          <td>1.879438</td>
          <td>300.002981</td>
        </tr>
        <tr>
          <th>21</th>
          <td>RP60</td>
          <td>4.483579e-02</td>
          <td>3.014400e+08</td>
          <td>4.481243e-02</td>
          <td>4.485916e-02</td>
          <td>0.000266</td>
          <td>2.575352</td>
          <td>300.002735</td>
        </tr>
        <tr>
          <th>22</th>
          <td>RP77</td>
          <td>2.711905e-07</td>
          <td>1.342230e+09</td>
          <td>2.433297e-07</td>
          <td>2.990513e-07</td>
          <td>0.052417</td>
          <td>0.280529</td>
          <td>300.000783</td>
        </tr>
        <tr>
          <th>23</th>
          <td>Four-branch serial system</td>
          <td>2.225032e-03</td>
          <td>1.351070e+09</td>
          <td>2.222519e-03</td>
          <td>2.227545e-03</td>
          <td>0.000576</td>
          <td>2.239469</td>
          <td>300.000255</td>
        </tr>
        <tr>
          <th>24</th>
          <td>R-S</td>
          <td>7.864349e-02</td>
          <td>1.839670e+09</td>
          <td>7.863119e-02</td>
          <td>7.865579e-02</td>
          <td>0.000080</td>
          <td>3.097966</td>
          <td>300.000382</td>
        </tr>
        <tr>
          <th>25</th>
          <td>Axial stressed beam</td>
          <td>2.919903e-02</td>
          <td>1.391830e+09</td>
          <td>2.919019e-02</td>
          <td>2.920788e-02</td>
          <td>0.000155</td>
          <td>2.810891</td>
          <td>300.001095</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 37-40

.. code-block:: Python

    data = df.values









.. GENERATED FROM PYTHON SOURCE LINES 41-46

.. code-block:: Python

    pf_reference = data[:, 1]
    pmin = data[:, 3]
    pmax = data[:, 4]









.. GENERATED FROM PYTHON SOURCE LINES 47-52

.. code-block:: Python

    benchmarkProblemList = otb.ReliabilityBenchmarkProblemList()
    numberOfProblems = len(benchmarkProblemList)
    numberOfProblems






.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    26



.. GENERATED FROM PYTHON SOURCE LINES 53-56

.. code-block:: Python

    digitsMinimum = 2









.. GENERATED FROM PYTHON SOURCE LINES 57-63

.. code-block:: Python

    categoryA = []
    categoryB = []
    categoryC = []
    categoryD = []









.. GENERATED FROM PYTHON SOURCE LINES 64-90

.. code-block:: Python

    for i in range(numberOfProblems):
        problem = benchmarkProblemList[i]
        name = problem.getName()
        pf = problem.getProbability()
        event = problem.getEvent()
        antecedent = event.getAntecedent()
        distribution = antecedent.getDistribution()
        dimension = distribution.getDimension()
        if pf > pmin[i] and pf < pmax[i]:
            tagBounds = "In"
        else:
            tagBounds = "Out"
        digits = otb.ComputeLogRelativeError(pf_reference[i], pf)
        if tagBounds == "In" and digits >= digitsMinimum:
            categoryA.append(name)
        elif tagBounds == "Out" and digits >= digitsMinimum:
            categoryB.append(name)
        elif tagBounds == "In" and digits < digitsMinimum:
            categoryC.append(name)
        else:
            categoryD.append(name)
        print(
            "#%d, %-10s, pf=%.2e, ref=%.2e, C.I.=[%.2e,%.2e], digits=%d : %s"
            % (i, name[0:10], pf, pf_reference[i], pmin[i], pmax[i], digits, tagBounds)
        )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    #0, RP8       , pf=7.90e-04, ref=7.91e-04, C.I.=[7.87e-04,7.94e-04], digits=2 : In
    #1, RP14      , pf=7.73e-04, ref=7.71e-04, C.I.=[7.69e-04,7.73e-04], digits=2 : In
    #2, RP22      , pf=4.21e-03, ref=4.21e-03, C.I.=[4.20e-03,4.21e-03], digits=4 : In
    #3, RP24      , pf=2.86e-03, ref=2.86e-03, C.I.=[2.86e-03,2.86e-03], digits=3 : In
    #4, RP25      , pf=4.15e-05, ref=4.18e-05, C.I.=[4.14e-05,4.21e-05], digits=2 : In
    #5, RP28      , pf=1.45e-07, ref=1.32e-07, C.I.=[1.15e-07,1.48e-07], digits=0 : In
    #6, RP31      , pf=3.23e-03, ref=3.23e-03, C.I.=[3.22e-03,3.23e-03], digits=3 : In
    #7, RP33      , pf=2.57e-03, ref=2.57e-03, C.I.=[2.57e-03,2.58e-03], digits=2 : Out
    #8, RP35      , pf=3.48e-03, ref=3.48e-03, C.I.=[3.48e-03,3.48e-03], digits=5 : In
    #9, RP38      , pf=8.10e-03, ref=8.06e-03, C.I.=[8.05e-03,8.07e-03], digits=2 : Out
    #10, RP53      , pf=3.13e-02, ref=3.13e-02, C.I.=[3.13e-02,3.13e-02], digits=3 : Out
    #11, RP55      , pf=5.60e-01, ref=5.60e-01, C.I.=[5.60e-01,5.60e-01], digits=4 : In
    #12, RP54      , pf=9.98e-04, ref=9.93e-04, C.I.=[9.88e-04,9.97e-04], digits=2 : Out
    #13, RP57      , pf=2.84e-02, ref=2.82e-02, C.I.=[2.82e-02,2.82e-02], digits=2 : Out
    #14, RP75      , pf=9.82e-03, ref=9.82e-03, C.I.=[9.81e-03,9.82e-03], digits=4 : In
    #15, RP89      , pf=5.43e-03, ref=5.47e-03, C.I.=[5.47e-03,5.47e-03], digits=2 : Out
    #16, RP107     , pf=2.92e-07, ref=2.75e-07, C.I.=[2.30e-07,3.20e-07], digits=1 : In
    #17, RP110     , pf=3.19e-05, ref=3.18e-05, C.I.=[3.15e-05,3.21e-05], digits=2 : In
    #18, RP111     , pf=7.65e-07, ref=7.85e-07, C.I.=[7.41e-07,8.29e-07], digits=1 : In
    #19, RP63      , pf=3.79e-04, ref=3.77e-04, C.I.=[3.72e-04,3.82e-04], digits=2 : In
    #20, RP91      , pf=6.97e-04, ref=7.00e-04, C.I.=[6.98e-04,7.02e-04], digits=2 : Out
    #21, RP60      , pf=4.56e-02, ref=4.48e-02, C.I.=[4.48e-02,4.49e-02], digits=1 : Out
    #22, RP77      , pf=2.87e-07, ref=2.71e-07, C.I.=[2.43e-07,2.99e-07], digits=1 : In
    #23, Four-branc, pf=2.22e-03, ref=2.23e-03, C.I.=[2.22e-03,2.23e-03], digits=2 : In
    #24, R-S       , pf=7.86e-02, ref=7.86e-02, C.I.=[7.86e-02,7.87e-02], digits=4 : In
    #25, Axial stre, pf=2.92e-02, ref=2.92e-02, C.I.=[2.92e-02,2.92e-02], digits=4 : In




.. GENERATED FROM PYTHON SOURCE LINES 91-101

There are four different cases.

* Category A: all good. For some problems, both metrics are correct in the sense
  that the reference probability is within the bounds and the number of significant digits is larger than 2.
  The RP24, RP55, RP110, RP63, R-S, Axial stressed beam problems fall in that category.
* Category B: correct digits, not in bounds.
  We see that the RP8 problem has a reference probability outside of the 95% confidence bounds,
  but has 2 significant digits.
* Category C: insufficient digits, in bounds. The difficult RP28 problem fall in that category.
* Category D: insufficient digits, not in bounds. These are suspicious problems.

.. GENERATED FROM PYTHON SOURCE LINES 103-105

.. code-block:: Python

    print(categoryA)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['RP8', 'RP14', 'RP22', 'RP24', 'RP25', 'RP31', 'RP35', 'RP55', 'RP75', 'RP110', 'RP63', 'Four-branch serial system', 'R-S', 'Axial stressed beam']




.. GENERATED FROM PYTHON SOURCE LINES 106-108

.. code-block:: Python

    print(categoryB)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['RP33', 'RP38', 'RP53', 'RP54', 'RP57', 'RP89', 'RP91']




.. GENERATED FROM PYTHON SOURCE LINES 109-111

.. code-block:: Python

    print(categoryC)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['RP28', 'RP107', 'RP111', 'RP77']




.. GENERATED FROM PYTHON SOURCE LINES 112-114

.. code-block:: Python

    print(categoryD)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['RP60']




.. GENERATED FROM PYTHON SOURCE LINES 115-118

The number of suspicious problems seems very large.
However, we notice that all these cases are so that the reference probability is close,
in absolute value, to the Monte-Carlo estimator.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.430 seconds)


.. _sphx_glr_download_auto_examples_plot_check_reliability_reference_probabilities.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_check_reliability_reference_probabilities.ipynb <plot_check_reliability_reference_probabilities.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_check_reliability_reference_probabilities.py <plot_check_reliability_reference_probabilities.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_check_reliability_reference_probabilities.zip <plot_check_reliability_reference_probabilities.zip>`