Optimization Methods

We now consider a generic unconstrained function-modeling (optimization) problem, applicable, for example, to classification problems in computer vision. The considerations presented in this section apply to the least-squares case but can be extended to a generic loss function.

Let $\mathbf{z}$ be the set of data involved in the modeling operation, consisting of a pair $(\mathbf{x}_i,y_i)$ composed of an arbitrary input $\mathbf{x}_i$ and the output $y_i$. Let $\ell (\hat{y}, y)$ be the cost function (loss function) that returns the quality of the estimate on $y$. The goal is to find the weights $\boldsymbol\beta$ that parameterize the function $f(\mathbf{x}; \boldsymbol\beta)$ and minimize a cost function $S(\boldsymbol\beta)$

$$S( \boldsymbol\beta) = \int \ell(\mathbf{z} ; \boldsymbol\b... ...S( \boldsymbol\beta) = \sum_{i=1}^{n} \ell_i(\boldsymbol\beta)$$ (4.27)

both in the continuous and discrete cases, having defined $\ell_i(\boldsymbol\beta) = \ell (f_i(\mathbf{x}_i; \boldsymbol\beta), y_i )$. For simplicity, the second case, namely the discrete case, will always be used to describe the cost function.

In the case of additive normal Gaussian error, the maximum-likelihood estimator is the quadratic loss function of equation (4.6):

$$\ell_i(\boldsymbol\beta) = r_i^2 (\boldsymbol\beta) = \left( y_i - f_i(\mathbf{x}_i ; \boldsymbol\beta) \right)^2$$ (4.28)

In practical applications, it is almost never possible to obtain the minimum of the function in closed form. It is therefore necessary to use suitable iterative methods which, starting from an initial state and moving along appropriate directions $\boldsymbol\delta$, gradually approach the minimum of the objective function.