Regression with Anisotropic Noise

When the observation noise is not isotropic, the Euclidean distance can no longer be used to measure the error; instead, the Mahalanobis distance is required. Under this different metric, the cost function (4.46) is written as

$$S(\boldsymbol\beta) = \left( \mathbf{r}(\boldsymbol\beta) \... ... \boldsymbol\Omega \left( \mathbf{r}(\boldsymbol\beta) \right)$$ (4.65)

where $\boldsymbol\Omega = \boldsymbol\Sigma^{-1}$ is the information matrix, also called the concentration matrix or precision matrix. The Mahalanobis distance is the optimal estimator in the Maximum Likelihood sense when the noise is zero-mean anisotropic Gaussian noise. In the special case where the covariance matrix is diagonal, this approach reduces entirely to the weighted least-squares approach.

The Taylor expansion of equation (4.65) is written as

$$\begin{array}{rl} S(\boldsymbol\beta + \boldsymbol\delta) &... ...oldsymbol\delta^{\top} \mathbf{H} \boldsymbol\delta \end{array}$$ (4.66)

with $\mathbf{r}$ and $\mathbf{J}$ evaluated at $\boldsymbol\beta$. Matrix $\mathbf{H} = \mathbf{J}^{\top} \boldsymbol\Omega \mathbf{J}$ is the information matrix of the entire system, as it is obtained by projecting the measurement error into parameter space through the Jacobian $\mathbf{J}$, while $\mathbf{b} = \mathbf{r}^{\top} \boldsymbol\Omega \mathbf{J}$ was introduced for compactness.

The derivatives of function $S$ therefore become

$$\dfrac{\partial S(\boldsymbol\beta + \boldsymbol\delta)}{\p... ...l\delta} \approx 2 \mathbf{b} + 2 \mathbf{H} \boldsymbol\delta$$ (4.67)

From this result, if the minimum of the cost function $S$ is sought using Gauss-Newton, a result similar to that obtained earlier is obtained:

$$\mathbf{H} \boldsymbol\delta = - \mathbf{b}$$ (4.68)

a result very similar to that of equation (4.49), obtained by Gauss-Newton with isotropic noise.