Multiclass Classification

SVM returns an objective function $f(\mathbf{x}_i, \mathbf{W}, b) = \mathbf{W} \mathbf{x}_i + b$ whose absolute value lacks a true meaning as it is an uncalibrated output. The extension to the multiclass case is challenging because the different objective functions for each class are not directly comparable to one another.

The concept of hinge loss can, however, be extended to the multiclass case. In this scenario, a SVM Loss of the form

$$\ell_i = \sum_{j \neq y_i} \begin{cases} 0, & \text{if } s... ...\ \end{cases} = \sum_{j \neq y_i} \max (0, s_j - s_{y_i} +1 )$$ (4.41)

is defined, where $s_j = f_j(\mathbf{x}_i)$ denotes, for simplicity, the objective function associated with class $j$ for the $i$-th sample.

Another similar metric is the squared hinge loss:

$$\ell_i = \sum_{j \neq y_i} \max \left(0, s_j - s_{y_i} + 1 \right)^{2}$$ (4.42)

Finally, a loss function is defined over the entire dataset as the average

$$\mathcal{L} = \frac{1}{n} \sum^{n}_{i=1} \ell_i + \lambda R(\mathbf(W))$$ (4.43)

with the optional regularization term on the weights.

A different metric, extended to the multiclass case, is the normalized exponential function known as Softmax:

$$\ell_i = - \log \frac{e^{s_{y_i} }}{\sum_j e^{s_j} } = - s_{y_i} + \log \sum_j e^{s_j}$$ (4.44)

The objective function $s_j$ can be interpreted as an unnormalized logarithmic probability for each class, and therefore, the cardinal loss function can be replaced with the cross-entropy loss function. A Softmax classifier minimizes the cross-entropy among the classes, and since it minimizes the negative log likelihood of the correct class, it can be viewed as a maximum likelihood estimator. In the case of Softmax, the regularization term $R(\mathbf(W))$ can be seen, from a statistical perspective, as a prior on the weights: in this case, it represents a Maximum a posteriori (MAP) estimate.