Linear Regression with a Polynomial Function

The method used to obtain linear regression for a line expressed in explicit form can be generalized to any polynomial function of the form:

$$y = \beta_0 + \beta_1 x + \beta_2 x^2 + \ldots + \beta_m x^m + \varepsilon$$ (4.97)

where $\beta_0 \ldots \beta_m$ are the parameters of the curve to be estimated, obtained by minimizing the error function described in (4.6). The derivatives of a polynomial function are notable:

$$\begin{array}{rl} \frac{\partial S}{\partial \beta_j} & = ... ...+ \ldots + \beta_m \sum x_i^{j+m} - \sum y_i x_i^j \end{array}$$ (4.98)

Setting the gradient to zero therefore amounts to solving the associated system:

$$\begin{bmatrix} \sum 1 & \ldots & \sum x_i^{m} \\ \sum x... ... \sum y_i x_i \\ \vdots \\ \sum y_i x_i^m \\ \end{bmatrix}$$ (4.99)

which is a symmetric matrix.

Alternatively, the theory of the pseudoinverse can be used (Section 1.1), directly applying equation (4.97) to construct an overdetermined linear system:

$$\begin{bmatrix} 1 & x_1 & \ldots & x_1^{m} \\ 1 & x_2 & ... ...bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \\ \end{bmatrix}$$ (4.100)

the Vandermonde matrix. The solution of this system yields the coefficients of the polynomial that minimizes the squared residuals. If one considers the pseudoinverse solved using the normal equations method, it can be seen that the resulting system is exactly the same as that of equation (4.99).

As will be seen elsewhere in this book, matrices such as the Vandermonde matrix, whose columns have different orders of magnitude, are ill-conditioned and require normalization to improve their numerical stability.