Input-dependent discrepancies between Monte Carlo vs FORM probabilities

Python usage
1

I am performing a sensitivity analysis for probabilities of failure (POFs) using FORM and standard Monte Carlo sampling, and have noticed a significant discrepancy when the POFs drop below approximately 1e-4, the FORM and MC results no longer align. This occurs despite using 1e6 or even 1e7 Monte Carlo samples. At higher POFs, the results tend to align more closely. The limit state function and random variable distribution types are constant, with the only difference between parameter values of some of the input distributions.

image

What is the reason for POFS calculated by Monte Carlo scattering around just below 1e-4, despite the high number of Monte Carlo samples and small maximum allowable Monte Carlo CoV?

Any help is much appreciated!

The FORM setup is:

optimizer = ot.Cobyla()
n_eval=10000
eps = 1e-7
optimizer.setMaximumEvaluationNumber(n_eval)
optimizer.setMaximumAbsoluteError(eps)
optimizer.setMaximumRelativeError(eps)
optimizer.setMaximumResidualError(eps)
optimizer.setMaximumConstraintError(eps)
algo = ot.FORM(optimizer, my_event, my_random_vector.getMean())
algo.run()

And for Monte Carlo:

n_mc_iter = 500000
n_samples_per_iter = 20
mc_cv_error = 1e-6

experiment = ot.MonteCarloExperiment()
algo = ot.ProbabilitySimulationAlgorithm(my_event, experiment)
algo.setMaximumOuterSampling(n_mc_iter)
algo.setBlockSize(n_samples_per_iter)
algo.setMaximumCoefficientOfVariation(mc_cv_error)
algo.run()
3

Hi again,

What surprises me is the fact that the FORM approximation gives smaller and smaller probabilities while Monte Carlo gives an almost constant probability. Could you give more details on your case? At least the probabilistic model? It may be due to the isoprobabilistic transform, or a strange behavior of the function defining the failure domain, or a bad convergence of the optimization in the FORM algorithm. Could you add optimizer.setVerbose(True) and post the output in order to check if there is a problem in the optimization step?

Cheers

Régis

4

Hi Regis,

The limit state function is highly non-linear and contains discontinuities in the form of if-else statements that depend on the value of some input random variables (RVs). There are nine input variables in total. Unfortunately I cannot provide the code, since it is proprietary.

That said, I ran an experiment that varied two of the nine input RVs and found that FORM results begin to deviate substantially as one of the RVs, let’s call this X1, decreases in magnitude (which logically results in a smaller failure probability). There is also an interesting interaction effect when another RV, X2, decreases, which also increases the deviation, as shown in the figure:

image

Below are the last few iterations from the FORM optimizer for one of the results with PMC = 3e-4 (1 million samples) and PFORM = 2.3e-12. SORM probabilities are on the same order as FORM, so the deviation with MC persists.

cobyla: NFVALS =  495, F = 2.392334E+01, MAXCV = 5.511005E-07
cobyla: X =-6.119078E-01   3.878792E-02   2.161279E-01   3.284527E-01
cobyla:    -8.605537E-01   4.069823E-08  -5.709614E+00  -2.420911E+00
cobyla:     2.848703E+00
cobyla: NFVALS =  496, F = 2.392334E+01, MAXCV = 1.156170E-11
cobyla: X =-6.119078E-01   3.878796E-02   2.161280E-01   3.284526E-01
cobyla:    -8.605536E-01   5.580028E-08  -5.709614E+00  -2.420911E+00
cobyla:     2.848703E+00
cobyla: NFVALS =  497, F = 2.392334E+01, MAXCV = 7.117423E-06
cobyla: X =-6.119078E-01   3.878796E-02   2.161279E-01   3.284527E-01
cobyla:    -8.605537E-01   3.493492E-08  -5.709614E+00  -2.420911E+00
cobyla:     2.848703E+00
cobyla: NFVALS =  498, F = 2.392334E+01, MAXCV = 2.293038E-11
cobyla: X =-6.119078E-01   3.878801E-02   2.161279E-01   3.284527E-01
cobyla:    -8.605537E-01  -4.381935E-09  -5.709614E+00  -2.420911E+00
cobyla:     2.848703E+00
cobyla: NFVALS =  499, F = 2.392334E+01, MAXCV = 8.831997E-08
cobyla: X =-6.119078E-01   3.878791E-02   2.161280E-01   3.284526E-01
cobyla:    -8.605537E-01   3.592916E-08  -5.709614E+00  -2.420911E+00
cobyla:     2.848703E+00
cobyla: NFVALS =  500, F = 2.392334E+01, MAXCV = 3.944681E-12
cobyla: X =-6.119079E-01   3.878797E-02   2.161279E-01   3.284527E-01
cobyla:    -8.605537E-01   5.767060E-08  -5.709614E+00  -2.420911E+00
cobyla:     2.848703E+00
cobyla: normal return.
cobyla: NFVALS =  500, F = 2.392334E+01, MAXCV = 3.944681E-12
cobyla: X =-6.119079E-01   3.878797E-02   2.161279E-01   3.284527E-01
cobyla:    -8.605537E-01   5.767060E-08  -5.709614E+00  -2.420911E+00
cobyla:     2.848703E+00

Process finished with exit code 0
5

Just a general answer.
When I look at your result, POF = 3e-4 using 1M samples, I think 1M is too small.
If the true POF is very small, <0.1%, you will need much more Monte Carlo Simulation samples (nMCS) than 1e6 or 1e7.
Based on my experience.To validate POF_true = 0.1%, I used nMCS = 2e7. nMCS depends on the quality of your Pseudo-random number generator.

About FORM and SORM, if your function is highly-nonlinear, try a different method called DRM (Dimension Reduction Method).