The Cramér-Rao Lower Bound

The Cramér-Rao lower bound (Cramer-Rao Lower Bound, CRLB) establishes a lower bound on the variance of every unbiased estimator of the parameter $\theta$ (to maintain notation consistent with the literature, $\boldsymbol\beta$ in our case).

Let $X$ be a multidimensional random variable and $\theta$ an unknown deterministic parameter. Let $f(x;\theta)$ be the probability density of $X$ parameterized by $\theta$. We assume that this density exists, is sufficiently regular, and is twice differentiable with respect to $\theta$.

Theorem 1 (Cramér-Rao inequality)   Let $T(X)$ be an unbiased estimator of the scalar parameter $\theta$, that is, such that

$$\E_\theta[T(X)] = \theta ,$$

and suppose that the support of the distribution $f(x;\theta)$ does not depend on $\theta$. Then, under suitable regularity conditions,

$$\var _\theta[T(X)] \ge \frac{1}{I(\theta)}$$ (4.1)

where

$$I(\theta) = \E_\theta \left[ \left( \frac{\partial}{\partial \theta} \ln f(X;\theta) \right)^2 \right]$$ (4.2)

is the Fisher information associated with the parameter $\theta$.

Since an unbiased estimator satisfies

$$\var _\theta[T(X)] = \E_\theta\!\left[ \left(T(X)-\theta\right)^2 \right],$$

the Cramér-Rao inequality provides a lower bound on the variance of any unbiased estimator.

Since the parameter $\theta$ is unknown, the Cramér-Rao theorem is primarily useful for evaluating whether an estimator is efficient, that is, whether its variance approaches the theoretically attainable minimum.