Logistic Regression

Logistic regression is one of the simplest and most widely used linear classifiers in statistics and machine learning. Unlike LDA, which derives from probabilistic assumptions about class distributions, logistic regression directly models the posterior probability of the categories and is therefore classified as a discriminative method.

As with other linear classifiers, a discriminant function of the form is considered

$$f(\mathbf{x}) = \mathbf{w}^{\top}\mathbf{x} + b$$ (5.17)

where $\mathbf{w}$ is the weight vector and $b$ is the bias term.

In the binary case, assuming labels $y \in \{0,1\}$, the probability that sample $\mathbf {x}$ belongs to the positive class is modeled by the logistic function

$$p(y=1\vert\mathbf{x}) = \frac{1}{1+e^{-f(\mathbf{x})}} = \frac{1}{1+e^{-(\mathbf{w}^{\top}\mathbf{x}+b)}}$$ (5.18)

whereas the probability of the negative class is

$$p(y=0\vert\mathbf{x}) = 1-p(y=1\vert\mathbf{x}).$$ (5.19)

The logistic function transforms the real value of the discriminant function into a probability between $0$ and $1$. The separating surface between the two classes corresponds to the condition

$$p(y=1\vert\mathbf{x}) = \frac{1}{2}$$ (5.20)

that is,

$$\mathbf{w}^{\top}\mathbf{x}+b = 0,$$ (5.21)

which coincides with a linear hyperplane.

Training consists of estimating the parameters $\mathbf{w}$ and $b$ that maximize the likelihood of the observed data. Given a training set $\{(\mathbf{x}_i,y_i)\}$, the log-likelihood is written as

$$\log \mathcal{L}(\mathbf{w},b) = \sum_i \left[ y_i \log p_i + (1-y_i)\log (1-p_i) \right]$$ (5.22)

where

$$p_i = p(y_i=1\vert\mathbf{x}_i).$$ (5.23)

Maximizing the log-likelihood is equivalent to minimizing the cost function

$$J(\mathbf{w},b) = -\sum_i \left[ y_i \log p_i + (1-y_i)\log (1-p_i) \right]$$ (5.24)

known as the cross-entropy loss or log-loss.

Unlike LDA, logistic regression makes no assumptions about the class distributions and can therefore also be applied when the observations do not follow a Gaussian distribution. On the other hand, like all linear classifiers, it can correctly separate only classes that are linearly separable in feature space.

Logistic regression can be interpreted as a probabilistic version of linear classifiers. The value $f(\mathbf{x})$ represents the logarithm of the ratio between the probabilities of the two classes

$$f(\mathbf{x}) = \log \frac {p(y=1\vert\mathbf{x})} {p(y=0\vert\mathbf{x})}$$ (5.25)

and is often called the log-odds or logit.