Iteratively Reweighted Least Squares

A technique orthogonal to M-estimators is iteratively reweighted least squares (IRLS, Iteratively Reweighted Least Squares) (gre84). This technique estimates new weights at each iteration and uses them to obtain a new solution. It can be applied to both linear and nonlinear problems.

In the linear case, the objective is to minimize a cost function of the form

$$\Vert \mathbf{W} \mathbf{r} \Vert^2 = \sum_i w_i^2 r_i^2 = \mathbf{r}^{\top} \mathbf{W}^{\top} \mathbf{W} \mathbf{r}$$ (4.122)

where the matrix $\mathbf{W}$ is a diagonal matrix with the weights $w_i$ along its diagonal.

In the overdetermined case, this has the solution

$$\mathbf{x} = \left[ \mathbf{A}^{\top} \mathbf{W}^{\top} \mat... ...{-1} \mathbf{A}^{\top} \mathbf{W}^{\top} \mathbf{W} \mathbf{b}$$ (4.123)

The same approach is applied to nonlinear systems.