Orthogonal Distance Fit

When the error affects both axes (noise as a function of distance), the cost function $S$ that maximizes the likelihood is called the Orthogonal least-squares line fit. The error can be expressed using the distance between the point and the line, according to equation (1.67). The regression using this metric, therefore called Perpendicular Regression or Total least squares (see Section 4.2.2), is meaningful when both coordinates are affected by error, that is, when both are random variables. The amount of noise on the two components is assumed to be equal (for the more general case, see the discussion in Section 2.4). The error function $S$ to be minimized is the distance between the point and the line:

$$S = \frac{1}{2n}\sum_{i=1}^{n} { \frac{(a x_i + b y_i + c)^2}{a^2 + b^2} }$$ (4.95)

and the minimum is found at $\nabla S = 0$. It should be noted that, in the case of perpendicular distance, both a minimum and a maximum exist as solutions; consequently, there are two line values (orthogonal to each other), both of which solve the system.

From the partial derivative $\frac{ \partial S}{\partial c}=0$, it follows that the regression line passes through the centroid $(\bar{x},\bar{y})$ of the distribution, that is,

$$c = - a \bar{x} - b \bar{y}$$ (4.96)

where $\bar{x}$ and $\bar{y}$ are the means of the samples $x_i$ and $y_i$, respectively.

Using relation (4.96), the error function (4.95) can be written as:

$$S = \frac{a^2 \left(\bar{x^2} - \bar{x}^2 \right) + 2 ab \le... ... \right) + b^2 \left(\bar{y^2} - \bar{y}^2 \right)}{a^2 + b^2}$$ (4.97)

that is, after suitable substitutions $S_{xx} = \text{var}(x)$, $S_{yy} = \text{var}(y)$, and $S_{xy} = \text{cov}(x,y)$:

$$S = \frac{a^2 S_{xx} + 2 ab S_{xy} + b^2 S_{yy} }{a^2 + b^2}$$ (4.98)

which is more readily differentiated. Expression (4.98) for the error is not general; it applies only to lines passing through the centroid of the distribution. Since it is a homogeneous form, it is defined only up to a multiplicative factor; therefore, there is not a single solution but a relation linking the parameters. Excluding the cases $a=0$ and $b=0$, which must be handled separately, the constraint for finding the minimum/maximum has the form

$$(a^2 -b^{2}) S_{xy} + a b (S_{yy} - S_{xx}) = 0$$ (4.99)

which is the solution to the problem.

Finally, it should be noted that the same result is obtained much more simply by applying the SVD decomposition to the line equation. In the case of linear regression, the SVD decomposition minimizes both the algebraic and geometric error (the algebraic and geometric errors coincide when all noisy terms are confined to the constant term).