Empirical Risk

Given a loss function $L$, the expected risk of a classifier is defined as

$$R(f) = \mathbb{E}_{(\mathbf{x},y)} \left[ L(y,f(\mathbf{x})) \right].$$ (5.54)

Since the data distribution is generally unknown, in practice the risk is estimated from a finite set of $N$ training samples:

$$\widehat{R}(f) = \frac{1}{N} \sum_{i=1}^{N} L(y_i,f(\mathbf{x}_i)).$$ (5.55)

This quantity is called the empirical risk. The principle of Empirical Risk Minimization (ERM) therefore consists in finding the model that minimizes the empirical risk on the training data.

In practice, empirical risk minimization is often accompanied by a regularization term that limits the complexity of the model:

$$\min_{\mathbf{w},b} \frac{1}{N} \sum_{i=1}^{N} L(y_i,f(\mathbf{x}_i)) + \lambda \Omega(\mathbf{w}).$$ (5.56)

This formulation makes it possible to interpret the Soft Margin SVM introduced in Section 5.5 as a special case of empirical risk minimization with regularization.