PCE with discrete distribution

Hello everybody,

With one of my student we are in the process of building a PCE for a function of continuous and discrete distribution of parameters.

To put it simple we are trying to fit functions of the type (see provided script):

f(X)=x_1 (x_2 - 1)^2 + x_3 x_1 x_2 x_4

with

  • x_1, x_2 uniform continuous variables
  • x_3 and x_4 discrete variables taking the values -1 or +1 (each of them)

Our actual functions are more complex than this one.

One first build the base of the PCE then use an hyperbolic anisotropic enumeration function. The idea was to limit the order of the polynomials acting for x3 and x4 (per the documentation they are limited to order 1, Orthogonal polynomials — OpenTURNS 1.20 documentation, Krawtchouk polynomials from the Bernouilli distribution interpreted as a simple case of a Binomial distribution).

As one wants to catch the x_2^2 term we put a weight for the enumeration function. After some checks we have seen that the (infinite) basis along the x3, x4 axis have orthogonal polynoms up to order 1. Higher orders are present but with inf and nan values as expected from the recursion formula.

We modified the basis by forcing the enumeration to put in front the wanted terms with the getStrataCumulatedCardinal method up to a given order of the polynomials involving the discrete terms.

Then we applied the cleaning strategy. It is working but not everytime. Sometimes the solver indicates that we are trying to identifiy on an orthogonal basis with too high orders on the x3, x4 axis. We obtained this by playing with the number provided for the in the getStrataCumulatedCardinal method.

We tried, alternatively, to build the basis ourselves with chosen polynomials but it does not work as OT characterize our basis as non orthogonal. Even if we think that it is actual a proper basis.

Finally we build our metamodel as follow:
Building 4 functions for each combinations of (x3,x4) 2-plet: f(x1,x2,+1,+1)=f_++(x1,x2), etc …
Building 4 metamodels for (x1,x2) continuous variables
Regrouping everything, it works perfectly.

The solution is somewhat OK as it is a little bit inaesthetic, is it possible to perform the task without the trick of splitting our function in 4 pieces ?

Regards

import openturns as ot
import openturns.viewer as viewer
import numpy as np

inputs = ['x1', 'x2', 'x3', 'x4']
formula = ['x1 * (x2 - 1)^2 + x3 * x1 * x2 * x4']
model = ot.SymbolicFunction(inputs, formula)
distribution = ot.ComposedDistribution(
    [ot.Uniform(1, 10), ot.Uniform(1, 10), 2*ot.Bernoulli(0.5)-1, 2*ot.Bernoulli(0.5)-1])

# Training sample
sample_size = 1000
X = ot.LHSExperiment(distribution, sample_size).generate()
Y = model(X)

# Sobol model
model_sa = ot.SaltelliSensitivityAlgorithm(distribution, sample_size, model, True)

# Multivariate basis
dim = distribution.getDimension()
uni_basis = ot.PolynomialFamilyCollection(dim)

for i in range(dim):
    distrib = distribution.getMarginal(i)
    uni_basis[i] = ot.StandardDistributionPolynomialFactory(distrib)

weight = 5
enumerate_function = ot.HyperbolicAnisotropicEnumerateFunction([1, 1, weight, weight], 1)
multi_basis = ot.OrthogonalProductPolynomialFactory(uni_basis, enumerate_function)
p = enumerate_function.getStrataCumulatedCardinal(
    enumerate_function.inverse([1, 1, 1, 1]) + 1)

n = 9

truncature_strategy = ot.CleaningStrategy(multi_basis, p, n, 1e-3, True)
basis_factory = ot.LARS()  # Least angle regression
fit_algo = ot.CorrectedLeaveOneOut()
approx_algo = ot.LeastSquaresMetaModelSelectionFactory(basis_factory, fit_algo)
coef_eval_strategy = ot.LeastSquaresStrategy(
    ot.MonteCarloExperiment(sample_size), approx_algo)

# PCE metamodel
algo = ot.FunctionalChaosAlgorithm(
    X, Y, distribution, truncature_strategy, coef_eval_strategy)
algo.run()
result = algo.getResult()
metamodel = result.getMetaModel()
coefficients = result.getCoefficients()
for i, j in enumerate(result.getIndices()):
    indices = enumerate_function(j)
    degree = sum(indices)
    print(str(int(degree)) + ' ' + str(indices) + ' ' + str(coefficients[i]))

# Sobol metamodel
chaosSI = ot.FunctionalChaosSobolIndices(result)
metamodel_sa = ot.SaltelliSensitivityAlgorithm(
    distribution, sample_size, metamodel, True)

# Validation sample
X_val = distribution.getSample(sample_size)
Y_val = model(X_val)

# Validation
val = ot.MetaModelValidation(X_val, Y_val, metamodel)
Q2 = val.computePredictivityFactor()
graph = val.drawValidation()
graph.setTitle("Q² = %.2f%%" % (Q2*100))
view = viewer.View(graph)
graph = model_sa.draw()
graph.add(metamodel_sa.draw())
view = viewer.View(graph)

# fixed_index = [1,2,3]
# fixed_values = [2,1,0]
# model_parametric = ot.ParametricFunction(model, fixed_index, fixed_values)
# metamodel_parametric = ot.ParametricFunction(metamodel, fixed_index, fixed_values)
# graph = ot.Graph('', 'x1', 'y0', True, 'bottomright')
# model_curve = model_parametric.getMarginal(0).draw(0, 10).getDrawable(0)
# model_curve.setLegend('Model')
# metamodel_curve = metamodel_parametric.getMarginal(0).draw(0, 10).getDrawable(0)
# metamodel_curve.setLegend('Metamodel')
# metamodel_curve.setLineStyle('dashed')
# metamodel_curve.setColor('red')
# graph.add(model_curve)
# graph.add(metamodel_curve)
# viewer.View(graph)

Hi,
The function space spanned by a discrete distribution with a finite number n of modalities is of finite dimension n, so you cannot build an orthonormal basis containing more than n different functions. It is why you get NaNs and Infs asking for Krawtchouk polynomials of degree greater than 1 when using Bernoulli distributions. It should be detected by the StandardDistributionPolynomialFactory algorithm but it is not, which is a bug. We should also introduce new enumerate functions adapted to this situation, by allowing the user to give limits to the orders in given components.

In the mean time you can cheat a little bit by adding additional possible values for the discrete components, but associated to negligible weights. This way you increase the dimension of the associated functional space (avoiding the problems with the polynomial factory and the enumerate function) and the metamodel you get is essentially the optimal one:

import openturns as ot
import openturns.viewer as viewer
import numpy as np

# Model
inputs = ["x1", "x2", "x3", "x4"]
formula = ["x1 * (x2 - 1)^2 + x3 * x1 * x2 * x4"]
model = ot.SymbolicFunction(inputs, formula)
marginals = [
    ot.Uniform(1, 10),
    ot.Uniform(1, 10),
    2 * ot.Bernoulli() - 1.0,
    2 * ot.Bernoulli() - 1.0,
]
distribution = ot.ComposedDistribution(marginals)

# TRICK: don't use the distribution but a modification of it to define the
# polynomial basis
# N is chosen such that no polynomial of degree greater than N+1 will be built
N = 5
eps = 1e-12
discrete = ot.UserDefined(
    [[-1], [1]] + [[i] for i in range(2, N)], [0.5] * 2 + [eps] * (N - 2)
)

instrumental_marginals = [ot.Uniform(1, 10), ot.Uniform(1, 10), discrete, discrete]

# Create the orthogonal basis
dimension = len(instrumental_marginals)
enumerateFunction = ot.LinearEnumerateFunction(dimension)
productBasis = ot.OrthogonalProductPolynomialFactory(
    [ot.StandardDistributionPolynomialFactory(d) for d in instrumental_marginals],
    enumerateFunction,
)

# Training sample
sample_size = 1000
X = ot.LHSExperiment(distribution, sample_size).generate()
Y = model(X)

# Create the projection strategy
projectionStrategy = ot.LeastSquaresStrategy()

# Create the adaptive strategy
degree = 6
indexMax = enumerateFunction.getStrataCumulatedCardinal(degree)
basisDimension = enumerateFunction.getStrataCumulatedCardinal(degree // 2)
threshold = 1.0e-6
adaptiveStrategy = ot.CleaningStrategy(
    productBasis, indexMax, basisDimension, threshold, False
)
algo = ot.FunctionalChaosAlgorithm(
    X, Y, distribution, adaptiveStrategy, projectionStrategy
)
algo.run()
metamodel = algo.getResult().getMetaModel()

# We can check that the metamodel involves polynomials of degree greater than 1
# in the discrete variables, but with very small coefficients
print("#" * 50)
print("metamodel=", metamodel)

# It can be cleaned using one of the two following keys in ResourceMap.
# This one is for output dimension >= 2
ot.ResourceMap.SetAsScalar("DualLinearCombinationEvaluation-SmallCoefficient", eps)
# This one is for output dimension == 1
ot.ResourceMap.SetAsScalar("LinearCombinationEvaluation-SmallCoefficient", eps)
algo = ot.FunctionalChaosAlgorithm(
    X, Y, distribution, adaptiveStrategy, projectionStrategy
)
algo.run()
metamodel = algo.getResult().getMetaModel()
print("#" * 50)
print("metamodel=", metamodel)

# Now check the quality of the metamodel
test_size = 10000
Xtest = distribution.getSample(test_size)
Ytest = model(Xtest)

validation = ot.MetaModelValidation(Xtest, Ytest, metamodel)
print("Q2=", validation.computePredictivityFactor()[0])
ot.Show(validation.drawValidation())

You have a perfect recovery of your function.

Cheers

Régis

Hi Régis,

I have just come across this useful topic. I have tried to run your code and I this error comes up:

 productBasis = ot.OrthogonalProductPolynomialFactory(

  File ~\AppData\Local\miniforge3\envs\tilt-env\Lib\site-packages\openturns\orthogonalbasis.py:2964 in __init__
    _orthogonalbasis.OrthogonalProductPolynomialFactory_swiginit(self, _orthogonalbasis.new_OrthogonalProductPolynomialFactory(*args))

RuntimeError: NotYetImplementedException :  in LinearEnumerateFunction::setUpperBound

I have updated to the latest version of OpenTURNS but the error continues. Not sure if this a known issue or I am missing something.

Thanks a lot

Best regards

Toni

The script should be updated as follows:

import openturns as ot
import openturns.viewer as viewer
import numpy as np

# Model
inputs = ["x1", "x2", "x3", "x4"]
formula = ["x1 * (x2 - 1)^2 + x3 * x1 * x2 * x4"]
model = ot.SymbolicFunction(inputs, formula)
marginals = [
    ot.Uniform(1, 10),
    ot.Uniform(1, 10),
    2 * ot.Bernoulli() - 1.0,
    2 * ot.Bernoulli() - 1.0,
]
distribution = ot.ComposedDistribution(marginals)

# TRICK: don't use the distribution but a modification of it to define the
# polynomial basis
# N is chosen such that no polynomial of degree greater than N+1 will be built
N = 5
eps = 1e-12
discrete = ot.UserDefined(
    [[-1], [1]] + [[i] for i in range(2, N)], [0.5] * 2 + [eps] * (N - 2)
)

instrumental_marginals = [ot.Uniform(1, 10), ot.Uniform(1, 10), discrete, discrete]

# Create the orthogonal basis
dimension = len(instrumental_marginals)
enumerateFunction = ot.HyperbolicAnisotropicEnumerateFunction(dimension, 1.0)
productBasis = ot.OrthogonalProductPolynomialFactory(
    [ot.StandardDistributionPolynomialFactory(d) for d in instrumental_marginals],
    enumerateFunction,
)

# Training sample
sample_size = 1000
X = ot.LHSExperiment(distribution, sample_size).generate()
Y = model(X)

# Create the projection strategy
projectionStrategy = ot.LeastSquaresStrategy()

# Create the adaptive strategy
degree = 6
indexMax = enumerateFunction.getStrataCumulatedCardinal(degree)
basisDimension = enumerateFunction.getStrataCumulatedCardinal(degree // 2)
threshold = 1.0e-6
adaptiveStrategy = ot.CleaningStrategy(
    productBasis, indexMax, basisDimension, threshold
)
algo = ot.FunctionalChaosAlgorithm(
    X, Y, distribution, adaptiveStrategy, projectionStrategy
)
algo.run()
metamodel = algo.getResult().getMetaModel()

# We can check that the metamodel involves polynomials of degree greater than 1
# in the discrete variables, but with very small coefficients
print("#" * 50)
print("metamodel=", metamodel)

# It can be cleaned using one of the two following keys in ResourceMap.
# This one is for output dimension >= 2
ot.ResourceMap.SetAsScalar("DualLinearCombinationEvaluation-SmallCoefficient", eps)
# This one is for output dimension == 1
ot.ResourceMap.SetAsScalar("LinearCombinationEvaluation-SmallCoefficient", eps)
algo = ot.FunctionalChaosAlgorithm(
    X, Y, distribution, adaptiveStrategy, projectionStrategy
)
algo.run()
metamodel = algo.getResult().getMetaModel()
print("#" * 50)
print("metamodel=", metamodel)

# Now check the quality of the metamodel
test_size = 10000
Xtest = distribution.getSample(test_size)
Ytest = model(Xtest)

validation = ot.MetaModelValidation(Ytest, metamodel(Xtest))
print("Q2=", validation.computeR2Score()[0])
ot.Show(validation.drawValidation())
1 Like

Thanks a lot.

Very useful to get hold of the different enumeration formulae.

Best

Toni