Moore–Penrose pseudoinverse

The solution (1.6) suggests introducing a matrix that, when applied to the vector $\mathbf{y}$, directly yields the least-squares solution. In the full-column-rank case, this matrix is

$$\left(\mathbf{A}^{\top}\mathbf{A}\right)^{-1}\mathbf{A}^{\top}.$$ (1.8)

However, this expression does not represent the general definition of the pseudoinverse, since it requires $\mathbf{A}^{\top}\mathbf{A}$ to be invertible.

The general definition of the Moore–Penrose pseudoinverse is naturally obtained through the singular value decomposition (Singular Value Decomposition, SVD). Let

$$\mathbf{A} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^{\top}$$ (1.9)

be the singular value decomposition of $\mathbf{A}$, where, in the reduced form for $m\geq n$,

$$\mathbf{U}\in\mathbb{R}^{m\times n}, \qquad \mathbf{\Sigma}\... ...bb{R}^{n\times n}, \qquad \mathbf{V}\in\mathbb{R}^{n\times n},$$

with orthonormal columns for $\mathbf{U}$ and $\mathbf{V}$ and with $\mathbf{\Sigma}$ diagonal:

$$\mathbf{\Sigma} = \operatorname{diag} \left( \sigma_1,\sigma... ...ight), \qquad \sigma_1\geq\sigma_2\geq\ldots\geq\sigma_n\geq0.$$ (1.10)

The values $\sigma_i$ are called the singular values of $\mathbf{A}$ and are related to the eigenvalues of the matrices $\mathbf{A}^{\top}\mathbf{A}$ and $\mathbf{A}\mathbf{A}^{\top}$ by the relation

$$\sigma_i^2=\lambda_i.$$ (1.11)

The Moore–Penrose pseudoinverse of $\mathbf{A}$ is defined as

$$\mathbf{A}^{+} = \mathbf{V}\mathbf{\Sigma}^{+}\mathbf{U}^{\top},$$ (1.12)

where $\mathbf{\Sigma}^{+}$ is obtained by replacing every nonzero singular value with its reciprocal and leaving the zero values unchanged:

$$\left(\mathbf{\Sigma}^{+}\right)_{ii} = \begin{cases} \dfrac... ...t{se }\sigma_i>0,\\ [6pt] 0 & \text{se }\sigma_i=0. \end{cases}$$ (1.13)

This definition is general, and the Moore–Penrose pseudoinverse exists and is unique for any matrix, even when $\mathbf{A}$ does not have full rank. The least-squares solution can therefore be expressed compactly as

$$\hat{\mathbf{x}} = \mathbf{A}^{+}\mathbf{y}.$$ (1.14)

When the system admits multiple least-squares solutions, $\mathbf{A}^{+}\mathbf{y}$ also returns the minimum-norm solution.

In the special case in which $\mathbf{A}$ has full column rank, definition (1.12) coincides with the expression obtained from the normal equations. In fact:

$$\begin{split} \mathbf{A}^{\top}\mathbf{A} &= \mathbf{V}\math... ... &= \mathbf{V}\mathbf{\Sigma}^{2}\mathbf{V}^{\top}, \end{split}$$ (1.15)

and therefore

$$\begin{split} \left(\mathbf{A}^{\top}\mathbf{A}\right)^{-1}\... ...\Sigma}^{-1}\mathbf{U}^{\top}\\ &= \mathbf{A}^{+}. \end{split}$$ (1.16)

Consequently,

$$\boxed{ \mathbf{A}^{+} = \left(\mathbf{A}^{\top}\mathbf{A}\right)^{-1} \mathbf{A}^{\top} }$$ (1.17)

represents only the special case of the pseudoinverse for matrices $\mathbf{A}$ with full column rank.