Total Least Squares

We now extend the linear problem $\mathbf{A}\mathbf{x}=\mathbf{b}+\delta$ to the more general case in which the coefficient matrix $\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{E}$ is also perturbed (Errors-In-Variables model, EIV (VHV91)). This type of least-squares regression problem is called Total Least Squares (TLS).

The solution of the perturbed system

$$(\mathbf{A} + \mathbf{E}) \mathbf{x} = \mathbf{b} + \boldsymbol\delta$$ (4.21)

corresponds to finding the solution $\mathbf {x}$ that minimizes the Frobenius norm $\Vert ( \mathbf{E} ~ \boldsymbol\delta ) \Vert _F$, subject to constraint (4.21). In classical TLS, all columns of the data matrix contain noise. If some columns are error-free, the solution is called mixed TLS-LS.

System (4.21) can be rewritten as

$$\left( \left[ \mathbf{A}\vert\mathbf{b} \right] + \left[ \ma... ...\begin{bmatrix} \mathbf{x} \\ -1 \end{bmatrix} = \mathbf{0}$$ (4.22)

By exploiting the SVD decomposition and the Eckart-Young-Mirsky theorem (the matrix formed by the first $n$ terms of the SVD decomposition is the matrix that best approximates matrix $\mathbf{Z}$ under the Frobenius norm), it is possible to find the solution to problem (4.21). Let

$$\mathbf{C} := \left[ \mathbf{A}\vert\mathbf{b} \right] = \mathbf{U} \boldsymbol\Sigma \mathbf{V}^{\top}$$ (4.23)

be the Singular Value Decomposition of matrix $\mathbf{C}$, where $\boldsymbol\Sigma = \diag \left(\sigma_1 \ldots \sigma_{n+d}\right)$. The Total Least Squares solution, if it exists, is written as

$$\hat{\mathbf{X}}_{tls} = - \mathbf{V}_{12} \mathbf{V}_{22}^{-1}$$ (4.24)

after partitioning

$$\mathbf{V}=\begin{bmatrix} \mathbf{V}_{11} & \mathbf{V}_{12... ...thbf{0} \\ \mathbf{0} & \boldsymbol\Sigma_2 \\ \end{bmatrix}$$ (4.25)

and the best estimate of $\hat{\mathbf{C}}$ can be obtained as

$$\hat{\mathbf{C}}_{tls} = \mathbf{C} + \Delta \mathbf{C}_{tl... ...} \diag \left(\boldsymbol\Sigma_1, 0 \right) \mathbf{V}^{\top}$$ (4.26)