Elements of Probability

This section presents some probability relations that will be useful in the following section.

In the discrete case, we define the probability density function (probability density function, PDF) as

$$p_X (x) = P(X = x)$$ (2.78)

whereas in the continuous case $p_X(x)$ denotes the probability density.

Bayes' theorem (or Bayes' formula) is a relation obtained by combining the compound-probability theorem with the total-probability theorem.

Starting from the definition of conditional probability $P(A,B)=P(A\vert B)P(B)$ (multiplication rule), we obtain:

$$P(A\vert B) = \frac{P(A,B)}{P(B)}$$ (2.79)

and conversely

$$P(B\vert A) = \frac{P(B,A)}{P(A)}$$ (2.80)

noting that $P(A,B)=P(B,A)$ yields

$$P(A\vert B) = \frac{P(B\vert A)P(A)}{P(B)}$$ (2.81)

The same reasoning can be applied to the case of three variables:

$$P(A,B,C) = P(A\vert B,C)P(B,C)=P(B\vert A,C)P(A,C)=P(B,A,C)$$ (2.82)

yielding Bayes' formula

$$P(A\vert B,C) = \frac{P(B\vert A,C)P(A\vert C)}{P(B\vert C)}$$ (2.83)

where the propagation of the dependence on a third variable $C$ can be seen.

Another important formula that will be used in the next chapter is the law of total probability (law of total probability):

$$P(B)=\sum P(A_i, B) = \sum P(A_i) P(B\vert A_i)$$ (2.84)

or, in the continuous case,

$$p_X(x)=\int p_{X,Y}(X=x,Y=y) dy=\int p(x\vert Y=y) p(y) dy$$ (2.85)

the marginal density of $\mathbf{X}$.