M-Estimator

The use of least-squares regression (Least squares) of the error rather than other weighting functions is motivated both by the maximum-likelihood function and, above all, by the simplicity of the derivatives obtained in the Jacobian.

In the problems considered so far, the variance has almost always been assumed to be constant, with observation errors distributed according to a normal distribution. If the noise were purely Gaussian, this approach would be theoretically correct. However, real applications generally involve distributions consisting of Gaussian noise associated with the model and noise associated with elements that do not belong to the model itself (outliers). Under these conditions, least-squares regression treats all points as if their errors were Gaussian: points close to the model are assigned little weight, whereas points far from the model are assigned substantial weight and, purely on probabilistic grounds, are usually outliers.

A unified way of dealing with these problems was introduced by John Nelder, who named these techniques generalized linear models (GLM, General Linear Models).

To solve this problem, the metric used to evaluate the errors must be changed: an initial example of a different metric that could solve the problem is absolute-value regression. However, finding the minimum of the error function expressed as an absolute-value distance (Least absolute deviations regression) is not easy, because its derivative is not continuous and requires iterative optimization techniques; differentiable metrics are preferable in this case.

In 1964, Peter Huber proposed a generalization of the maximum-likelihood minimization concept by introducing M-estimators.

Some examples of regression functions are shown in Figure 4.3.

Figure 4.3: Some examples of weighting functions for regression: least-squares regression (L2 metric), linear regression (L1), Huber estimators, Tukey's biweight, the Lorentzian (Cauchy) function, and the Welsch (Leclerc) function.

An M-estimator replaces the metric based on the sum of squares with a metric based on a generic $\rho$ function (loss function) having a unique minimum at zero and subquadratic growth. M-estimators generalize least-squares regression: setting $\rho(\mathbf{r})=\Vert\mathbf{r}\Vert^2$ yields the standard form of regression.

Finally, if the loss function is monotonically increasing, it is called an M-estimator; if the loss function increases near zero but decreases far from zero, it is called a redescending M-estimator.

The parameters are estimated by minimizing a sum of generic weighted quantities:

$$\min_\beta \sum \rho \left( \frac{\mathbf{r}_i}{\sigma_i} \right)$$ (4.120)

whose closed-form or iterative solution differs from that of least squares because of the different derivative of the function $\rho$:

$$\sum_{i=1}^{n} \frac{1}{\sigma_i} \rho' \left( \frac{\mathbf... ...artial \mathbf{r}_i}{\partial \beta_j} = 0 \qquad j=1,\ldots,m$$ (4.121)