Mean and Variance

It is easy to assume that the notion of the mean of a set of numbers is familiar to everyone, at least from a purely intuitive point of view. Nevertheless, this section provides a brief summary, gives the relevant definitions, and highlights some interesting aspects.

For $n$ samples of an observed quantity $x$, the sample mean sample mean is denoted by $\bar{x}$ and is given by

$$\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i$$ (2.1)

By definition, the sample mean is an empirical quantity.

If infinitely many values of $x$ could be sampled, $\bar{x}$ would converge to the theoretical expected value (expected value). This is the law of large numbers (Law of Large Numbers).

The expected value (expectation, mean) of a random variable $X$ is denoted by $\E[X]$ or $\mu$ and can be computed for discrete random variables using

$$\E[X] = \mu_x = \sum_{-\infty}^{+\infty} x_i p_X(x_i)$$ (2.2)

and for continuous variables using

$$\E[X] = \mu_x = \int_{-\infty}^{+\infty} x p_X(x) dx$$ (2.3)

given knowledge of the probability distribution $p_X(x)$.

We now introduce the concept of the mean of a function of a random variable.

Definizione 5   Let $X$ be a random variable and let $g(x)$ be a measurable function. In the discrete case, the expected value of the random variable $Y=g(X)$ is defined as

$$\E[g(X)] = \sum_i g(x_i) p_X(x_i),$$ (2.4)

whereas in the continuous case it is defined as

$$\E[g(X)] = \int_{-\infty}^{+\infty} g(x) p_X(x)\,dx.$$ (2.5)

Some functions have means with particularly significant interpretations. When $g(x)=x$, we speak of the first-order moment (first statistical moment), and, in general, when $g(x)=x^{k}$, we speak of $k$-order moments. The mean is therefore the first-order statistic, while another statistic of particular interest is the second-order moment:

$$\E[X^{2}] = \int_{-\infty}^{+\infty} x^{2} p_X(x) dx$$ (2.6)

This statistic is important because it allows the variance of $X$ to be estimated.

The variance is defined as the expected value of the square of the random variable $X$ after subtracting its mean, that is, as the second-order moment of the function $g(X)= X-\E[X]$:

$$\text{var}(X) = \sigma^{2}_X = \E[ (X - \E[X])^{2} ]$$ (2.7)

Since $\E[X]$ is a constant, expanding the square gives the equivalent and widely used form

$$\text{var}(X) = \sigma^{2}_X = \E[X^{2}] - \E[X]^{2}$$ (2.8)

The square root of the variance is known as the standard deviation (standard deviation) and has the advantage of having the same unit of measurement as the observed quantity:

$$\sigma_X = \sqrt{ \text{var}(X) }$$ (2.9)

We now extend the concepts introduced so far to the multivariate case. The multivariate case can be viewed as an extension to multiple dimensions, with a different variable associated with each dimension.

The covariance matrix $\Sigma$ is the multidimensional (or multivariable) extension of the concept of variance. It is constructed as

$$\Sigma_{ij} =\text{cov}(X_i,X_j)$$ (2.10)

where each matrix element contains the covariance between the various components of the random vector $X$. The covariance indicates how the different random variables composing the vector $X$ are related to one another.

The possible notations for the covariance matrix are

$$\Sigma = \E \left[ (X - \E[X])(X - \E[X])^{\top} \right] = \text{var}(X) = \text{cov}(X) = \text{cov}(X,X)$$ (2.11)

The notation for the cross-covariance is instead unique:

$$\text{cov}(X,Y) = \E \left[ (X - \E[X])(Y - \E[Y])^{\top} \right]$$ (2.12)

which generalizes the concept of the covariance matrix. The cross-covariance matrix $\boldsymbol\Sigma$ has, at position $(i,j)$, the covariance between the random variable $X_i$ and the variable $Y_j$:

$$\boldsymbol\Sigma = \begin{bmatrix} \text{cov}(X_1,Y_1) & \... ...cov}(X_1,Y_m) & \cdots & \text{cov}(X_n,Y_m) \\ \end{bmatrix}$$ (2.13)

Consequently, the covariance matrix $\text{cov}(X,X)$ is symmetric.

The covariance matrix describes how the different components are correlated with one another and is also called the scatter matrix (scatter matrix). The inverse of the covariance matrix is called the concentration matrix or precision matrix.

In the scalar case, the correlation coefficient between two random variables $X$ and $Y$ is defined as

$$r(X,Y) = \frac{\text{cov}(X,Y)} {\sqrt{\text{var}(X)\,\text{var}(Y)}} = \frac{\text{cov}(X,Y)} {\sigma_X \sigma_Y}.$$ (2.14)

The correlation coefficient takes values in the interval $[-1,1]$. Values close to $1$ indicate strong positive correlation, values close to $-1$ indicate strong negative correlation, whereas values close to $0$ indicate the absence of linear correlation ^2.1.

In the multivariate case, a correlation matrix is defined analogously by normalizing the cross-covariance matrix so that each element represents the correlation coefficient between a component of $X$ and a component of $Y$. In this case as well, all matrix elements belong to the interval $[-1,1]$.

Footnotes

... correlation^2.1

A zero correlation coefficient implies the absence of linear correlation but does not necessarily imply statistical independence.