Optimization Methods

We now consider a generic unconstrained function-modeling (optimization) problem, applicable, for example, to classification problems in computer vision. The considerations presented in this section apply to least-squares problems but can be extended to a generic loss function.

Let $\mathbf{z}$ be the set of data involved in the modeling operation, consisting of pairs $(\mathbf{x}_i,y_i)$ composed of an arbitrary input $\mathbf{x}_i$ and output $y_i$. Let $\ell (\hat{y}, y)$ be the cost function (loss function) that returns the quality of the estimate for $y$. The goal is to find the weights $\boldsymbol\beta$ that parameterize the function $f(\mathbf{x}; \boldsymbol\beta)$ and minimize a cost function $S(\boldsymbol\beta)$

$$S( \boldsymbol\beta) = \int \ell(\mathbf{z} ; \boldsymbol\b... ...S( \boldsymbol\beta) = \sum_{i=1}^{n} \ell_i(\boldsymbol\beta)$$ (4.30)

both in the continuous and discrete cases, having defined $\ell_i(\boldsymbol\beta) = \ell (f_i(\mathbf{x}_i; \boldsymbol\beta), y_i )$. For simplicity, we will always refer to the latter case, the discrete one, when describing the cost function.

In the case of additive Gaussian noise, the maximum-likelihood estimator is the quadratic loss function in Equation (4.7):

$$\ell_i(\boldsymbol\beta) = r_i^2 (\boldsymbol\beta) = \left( y_i - f_i(\mathbf{x}_i ; \boldsymbol\beta) \right)^2$$ (4.31)

In practical applications, it is almost never possible to obtain the minimum of the function in closed form. It is therefore necessary to use suitable iterative methods that start from an initial state and gradually approach the minimum of the objective function by moving along suitable directions $\boldsymbol\delta$.