Total Least Squares

We now extend the linear problem $\mathbf{A}\mathbf{x}=\mathbf{b}+\delta$ to the more general case where the coefficient matrix $\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{E}$ is also perturbed (Errors-In-Variables model, EIV (VHV91)). This type of least squares regression problem is referred to as Total Least Squares (TLS).

The solution of the perturbed system

$$(\mathbf{A} + \mathbf{E}) \mathbf{x} = \mathbf{b} + \boldsymbol\delta$$ (3.20)

corresponds to finding the solution $\mathbf {x}$ that minimizes the Frobenius norm $\Vert ( \mathbf{E} ~ \boldsymbol\delta ) \Vert _F$, subject to the constraint (3.20). In classical TLS, all columns of the data matrix contain noise. If some columns are error-free, then the solution is referred to as mixed TLS-LS.

The system (3.20) can be rewritten as

$$\left( \left[ \mathbf{A}\vert\mathbf{b} \right] + \left[ \ma... ...\begin{bmatrix} \mathbf{x} \\ -1 \end{bmatrix} = \mathbf{0}$$ (3.21)

Utilizing the SVD decomposition and the Eckart-Young-Mirsky theorem (the matrix formed by the first $n$ terms of the SVD decomposition is the matrix that best approximates the matrix $\mathbf{Z}$ under the Frobenius norm), it is possible to find the solution to the problem (3.20). Let us denote

$$\mathbf{C} := \left[ \mathbf{A}\vert\mathbf{b} \right] = \mathbf{U} \boldsymbol\Sigma \mathbf{V}^{\top}$$ (3.22)

the Singular Value Decomposition of the matrix $\mathbf{C}$, where $\boldsymbol\Sigma = \diag \left(\sigma_1 \ldots \sigma_{n+d}\right)$. The Total Least Squares solution, if it exists, can be expressed as

$$\hat{\mathbf{X}}_{tls} = - \mathbf{V}_{12} \mathbf{V}_{22}^{-1}$$ (3.23)

having partitioned

$$\mathbf{V}=\begin{bmatrix} \mathbf{V}_{11} & \mathbf{V}_{12... ...thbf{0} \\ \mathbf{0} & \boldsymbol\Sigma_2 \\ \end{bmatrix}$$ (3.24)

and it is possible to obtain the best estimate of $\hat{\mathbf{C}}$ as

$$\hat{\mathbf{C}}_{tls} = \mathbf{C} + \Delta \mathbf{C}_{tl... ...} \diag \left(\boldsymbol\Sigma_1, 0 \right) \mathbf{V}^{\top}$$ (3.25)