Orthogonal Distance Fit

When error is present along both axes (noise dependent on distance), the cost function $S$ that maximizes the likelihood is called the Orthogonal least-squares line fit. The error can in fact be expressed using the distance between the point and the line, according to equation (1.67). Regression using this metric, therefore called Perpendicular Regression or Total least squares (see Section 4.2.2), is meaningful when both coordinates are affected by error, that is, when both are random variables. The amount of noise on the two components is assumed to be equal (for the more general case, see the discussion in Section 2.4). The error function $S$ to be minimized is the distance between the point and the line:

$$S = \frac{1}{2n}\sum_{i=1}^{n} { \frac{(a x_i + b y_i + c)^2}{a^2 + b^2} }$$ (4.90)

and the minimum is found at $\nabla S = 0$. It should be noted that, in the case of perpendicular distance, both a minimum and a maximum exist as solutions; consequently, there are two line values, orthogonal to each other, that are both solutions of the system.

From the partial derivative $\frac{ \partial S}{\partial c}=0$, it follows that the regression line passes through the centroid $(\bar{x},\bar{y})$ of the distribution, namely

$$c = - a \bar{x} - b \bar{y}$$ (4.91)

where $\bar{x}$ and $\bar{y}$ are the means of the samples $x_i$ and $y_i$, respectively.

Using relation (4.91), the error function (4.90) can be written as:

$$S = \frac{a^2 \left(\bar{x^2} - \bar{x}^2 \right) + 2 ab \le... ... \right) + b^2 \left(\bar{y^2} - \bar{y}^2 \right)}{a^2 + b^2}$$ (4.92)

that is, after suitable substitutions $S_{xx} = \text{var}(x)$, $S_{yy} = \text{var}(y)$, and $S_{xy} = \text{cov}(x,y)$:

$$S = \frac{a^2 S_{xx} + 2 ab S_{xy} + b^2 S_{yy} }{a^2 + b^2}$$ (4.93)

which is easier to differentiate. Expression (4.93) for the error is not general, but applies only to all lines passing through the centroid of the distribution. Since it is a homogeneous form, it is defined only up to a multiplicative factor; therefore, there is no unique solution, but rather a relation linking the parameters. Excluding the cases $a=0$ and $b=0$, which must be treated separately, the constraint for obtaining the minimum/maximum has the form

$$(a^2 -b^{2}) S_{xy} + a b (S_{yy} - S_{xx}) = 0$$ (4.94)

which is the solution to the problem.

Finally, it should be noted that the same result is obtained much more simply by applying the SVD decomposition to the line equation. In the case of linear regression, the SVD decomposition minimizes both the algebraic and geometric errors (the algebraic and geometric errors coincide when all noise-affected terms remain confined to the constant term).