Uncertainty Propagation

Section 1.2 examined the sensitivity of a linear system to data perturbations through the condition number. This analysis considers the perturbation in deterministic terms, without specifying its statistical distribution.

In what follows, however, the error in the observations is assumed to be a random variable. In this case, it is not sufficient to determine how much a perturbation can modify the solution; the statistical characteristics of this variation must also be determined, for example through its variance or, in the multidimensional case, through its covariance matrix.

Consider therefore a linear transformation

$$\mathbf{y}=\mathbf{A}\mathbf{x}.$$ (2.28)

If $\mathbf {x}$ is a random variable with covariance matrix $\boldsymbol\Sigma_x$, the covariance of the variable $\mathbf{y}$ is

$$\boxed{ \boldsymbol\Sigma_y = \mathbf{A}\boldsymbol\Sigma_x\mathbf{A}^{\top} }.$$ (2.29)

This relation therefore describes the statistical propagation of uncertainty through the linear system.

In the case of an inverse linear system, with $\mathbf{A}$ square and invertible,

$$\mathbf{x}=\mathbf{A}^{-1}\mathbf{b},$$ (2.30)

one analogously obtains

$$\boxed{ \boldsymbol\Sigma_x = \mathbf{A}^{-1} \boldsymbol\Sigma_b \mathbf{A}^{-\top} }.$$ (2.31)

This relation highlights a complementary aspect with respect to the condition number. The condition number provides a bound on the relative amplification factor of a perturbation, whereas the covariance matrix describes how a specific error distribution is transformed by the system.

For an overdetermined system solved by least squares, assuming $\mathbf{A}$ known and of full column rank,

$$\hat{\mathbf{x}} = \mathbf{A}^{+}\mathbf{b},$$ (2.32)

and therefore, if uncertainty is present only in the vector $\mathbf{b}$,

$$\boxed{ \boldsymbol\Sigma_{\hat{x}} = \mathbf{A}^{+} \boldsymbol\Sigma_b (\mathbf{A}^{+})^{\top} }.$$ (2.33)

This relation establishes the connection between the theory of overdetermined linear systems and the propagation of statistical uncertainty.

Generalizing the linear case, consider an affine transformation

$$f(\mathbf{x}) = f(\bar{\mathbf{x}}) + \mathbf{A}(\mathbf{x}-\bar{\mathbf{x}}),$$ (2.34)

where $\bar{\mathbf{x}}=\E[\mathbf{x}]$. The random variable $Y=f(X)$ therefore has expected value

$$\bar{\mathbf{y}}=f(\bar{\mathbf{x}})$$ (2.35)

and covariance matrix

$$\boldsymbol\Sigma_y = \mathbf{A}\boldsymbol\Sigma_x\mathbf{A}^{\top}.$$ (2.36)

The affine case will be particularly useful for handling nonlinear transformations, which can be locally approximated by an affine transformation.

Nonlinear Transformations

Covariance propagation in the nonlinear case cannot in fact be readily obtained in closed form and generally must be computed approximately. Techniques such as Monte Carlo simulation can be used to approximate the probability distribution resulting from a generic nonlinear transformation.

The linear approximation is nevertheless widely used in practical problems. In particular, for first-order error propagation (first-order error propagation), the nonlinear transformation $f$ is approximated, through a Taylor-series expansion, by an affine transformation

$$f(\mathbf{x}) \approx f(\bar{\mathbf{x}}) + \mathbf{J}_f(\mathbf{x}-\bar{\mathbf{x}})$$ (2.37)

with $\mathbf{J}_f$ the matrix of partial derivatives (the Jacobian) of the function $f$, evaluated at point $\bar{\mathbf{x}}$.

With this approximation, the result of the affine case given previously in equation (2.34) can be used to determine the covariance matrix of the variable $f(\mathbf{x})$, replacing matrix $\mathbf{A}$ with the Jacobian, yielding

$$\boldsymbol\Sigma_y = \mathbf{J}_f \boldsymbol\Sigma_x \mathbf{J}_f^{\top}.$$ (2.38)

Under this approximation, the expected value of the transformed variable is assumed to be

$$\bar{\mathbf{y}} = f(\bar{\mathbf{x}}).$$ (2.39)

Local linearization also forms the basis of uncertainty propagation in the Extended Kalman Filter, whereas in sigma-point filters the nonlinear transformation is handled without explicitly computing the Jacobian.