Iteratively Reweighted Least Squares

A technique complementary to M-estimators is iteratively reweighted least squares (IRLS, Iteratively Reweighted Least Squares) (gre84). It is a technique in which new weights are estimated at each iteration and used to obtain a new solution. This technique can be applied to both linear and nonlinear problems.

In the linear case, the objective is to minimize a cost function of the form

$$\Vert \mathbf{W} \mathbf{r} \Vert^2 = \sum_i w_i^2 r_i^2 = \mathbf{r}^{\top} \mathbf{W}^{\top} \mathbf{W} \mathbf{r}$$ (4.127)

where the matrix $\mathbf{W}$ is a diagonal matrix with the weights $w_i$ along its diagonal.

In the overdetermined case, the solution is

$$\mathbf{x} = \left[ \mathbf{A}^{\top} \mathbf{W}^{\top} \mat... ...{-1} \mathbf{A}^{\top} \mathbf{W}^{\top} \mathbf{W} \mathbf{b}$$ (4.128)

The same approach is applied to nonlinear systems.