Regression with Anisotropic Noise

In the case where the observation noise is not isotropic, it is no longer possible to use the Euclidean distance to measure the error, and it is necessary to switch to the Mahalanobis distance. Under this different metric, the cost function (3.40) can be expressed as

$$S(\boldsymbol\beta) = \left( \mathbf{r}(\boldsymbol\beta) \... ... \boldsymbol\Omega \left( \mathbf{r}(\boldsymbol\beta) \right)$$ (3.57)

where $\boldsymbol\Omega = \boldsymbol\Sigma^{-1}$ is the information matrix, also known as the concentration matrix or precision matrix. The Mahalanobis distance is the optimal estimator in the sense of Maximum Likelihood when the noise is Gaussian and anisotropic with zero mean. In the particular case where the covariance matrix is diagonal, this approach can be completely reduced to the weighted least squares approach.

The Taylor series expansion of equation (3.57) is expressed as

$$\begin{array}{rl} S(\boldsymbol\beta + \boldsymbol\delta) &... ...oldsymbol\delta^{\top} \mathbf{H} \boldsymbol\delta \end{array}$$ (3.58)

with $\mathbf{r}$ and $\mathbf{J}$ calculated in $\boldsymbol\beta$. The matrix $\mathbf{H} = \mathbf{J}^{\top} \boldsymbol\Omega \mathbf{J}$ is the information matrix of the entire system, as it is obtained from the projection of the measurement error in the parameter space through the Jacobian $\mathbf{J}$, while $\mathbf{b} = \mathbf{r}^{\top} \boldsymbol\Omega \mathbf{J}$ has been introduced for compactness.

The derivatives of the function $S$ consequently become

$$\dfrac{\partial S(\boldsymbol\beta + \boldsymbol\delta)}{\p... ...l\delta} \approx 2 \mathbf{b} + 2 \mathbf{H} \boldsymbol\delta$$ (3.59)

From this result, if one wishes to find the minimum of the cost function $S$ using Gauss-Newton, a result similar to the one observed previously is obtained

$$\mathbf{H} \boldsymbol\delta = - \mathbf{b}$$ (3.60)

which is very similar to that of equation (3.43) obtained from Gauss-Newton with isotropic noise.