Elements of Probability

This section presents several probability relations that will be useful in the following section.

In the discrete case, we define the probability mass function (probability mass function, PMF) as

$$p_X(x) = P(X=x)$$ (2.82)

whereas, in the continuous case, $p_X(x)$ denotes the probability density function (probability density function, PDF).

Bayes' theorem (or Bayes' rule) is a relation obtained by combining the product rule of probability with the law of total probability.

Starting from the definition of conditional probability $P(A,B)=P(A\vert B)P(B)$ (multiplication rule), we obtain:

$$P(A\vert B) = \frac{P(A,B)}{P(B)}$$ (2.83)

and conversely

$$P(B\vert A) = \frac{P(B,A)}{P(A)}$$ (2.84)

noting that, since $P(A,B)=P(B,A)$, we obtain

$$P(A\vert B) = \frac{P(B\vert A)P(A)}{P(B)}$$ (2.85)

The same reasoning can be applied to the case of three variables:

$$P(A,B,C) = P(A\vert B,C)P(B,C)=P(B\vert A,C)P(A,C)=P(B,A,C)$$ (2.86)

which yields Bayes' formula

$$P(A\vert B,C) = \frac{P(B\vert A,C)P(A\vert C)}{P(B\vert C)}$$ (2.87)

where the propagation of the dependence on a third variable $C$ can be seen.

Another important formula that will be used in the next chapter is the law of total probability:

$$P(B)=\sum P(A_i, B) = \sum P(A_i) P(B\vert A_i)$$ (2.88)

or, in the continuous case,

$$p_X(x)=\int p_{X,Y}(X=x,Y=y) dy=\int p(x\vert Y=y) p(y) dy$$ (2.89)

the marginal density of $\mathbf{X}$.