Iteratively Reweighted Least Squares

An orthogonal technique to M-Estimators is the Iteratively Reweighted Least Squares (IRLS) method (gre84). This technique involves estimating new weights at each iteration, which are then used to derive a new solution. This method can be applied to both linear and non-linear problems.

In the linear case, the objective is to minimize a cost function of the form

$$\Vert \mathbf{W} \mathbf{r} \Vert^2 = \sum_i w_i^2 r_i^2 = \mathbf{r}^{\top} \mathbf{W}^{\top} \mathbf{W} \mathbf{r}$$ (3.105)

where the matrix $\mathbf{W}$ is a diagonal matrix with the weights $w_i$ placed along the diagonal.

In the case of an over-dimensioned problem, this has a solution

$$\mathbf{x} = \left[ \mathbf{A}^{\top} \mathbf{W}^{\top} \mat... ...{-1} \mathbf{A}^{\top} \mathbf{W}^{\top} \mathbf{W} \mathbf{b}$$ (3.106)

The same approach is applied to non-linear systems.