The Central Limit Theorem

Here is something that sounds suspiciously like a mathematical magic trick. Start with a population that does not resemble a bell curve at all. Select random samples, calculate their averages, and repeat. The distribution of those averages will often begin to resemble a bell curve.

That is the central limit theorem. And it is one of the reasons statistical inference is possible.

First, distinguish three things

1. The population distribution describes the individual values in the population.
2. A sample is a set of values drawn from that population.
3. The sampling distribution of the sample mean describes the means produced by repeatedly drawing samples of the same size.

The sampling distribution is not a histogram of all the individual observations. It is a histogram of a statistic computed once per sample.

This distinction matters. If a distribution of individual incomes is skewed, the distribution of averages across many independently selected groups can be far more symmetric.

The theorem (a useful version)

Suppose $X_1,\ldots,X_n$ are independent and identically distributed with mean $\mu$ and finite, positive variance $\sigma^2$. Then, as $n$ grows,

$$\frac{\bar X_n-\mu}{\sigma/\sqrt{n}} \xrightarrow{\ d\ } N(0,1).$$

In words: the standardized sample mean approaches the standard normal distribution.

It does not say that the original population becomes normal. It does not promise that a sample size of 30 is always sufficient. And it does not automatically apply to distributions without finite variance or to observations with unsuitable dependence.

The required sample size depends, among other things, on how skewed or heavy-tailed the population is.

Two discoveries at once

The theorem gives us a shape: approximate normality of the standardized sample mean for sufficiently large samples.

And the standard error,

$$\frac{\sigma}{\sqrt{n}},$$

tells us that the spread of sample means gets smaller with increasing sample size.

The mean of the sampling distribution remains at $\mu$. The estimates become more concentrated around it.

The sampling laboratory

Choose a population shape below, then choose a sample size. The laboratory will repeatedly draw samples and calculate a mean for each.

Start with a sample size of 1 and an exponential population. You should see a strongly skewed distribution. Change the size to 5, 20, and 80. The resulting distribution of sample means should get more bell-shaped and narrow.

Try the Bernoulli population next. With only two possible individual outcomes, how can averages produce so many possible values?

Every run is random, so the plots will differ. That is part of the point.

Check your understanding

1. Does the central limit theorem claim that individual observations must become normally distributed?
2. If the sample size increases by a factor of 9, what happens to the standard error under the standard assumptions?
3. Why is the rule “30 observations is always enough” unreliable?

Suggested responses

1. No: it describes the limiting distribution of a standardized average. 2. The standard error is divided by 3. 3. Rates of convergence depend on properties of the population, notably skewness and tails, among other assumptions.

The law of large numbers answers a different question

The weak law of large numbers says that $\bar X_n$ concentrates near $\mu$ in probability under suitable conditions. With independent finite-variance observations, Chebyshev’s inequality gives $P(|\bar X_n-\mu|\geq\varepsilon)\leq\sigma^2/(n\varepsilon^2)$. This bound tends to zero for any fixed positive tolerance. The CLT instead describes the shape of fluctuations after magnifying them by $\sqrt n$.

For independent Bernoulli observations with $p=0.2$ and $n=100$, the mean of the sample proportion is $0.2$ and its standard error is $\sqrt{0.2(0.8)/100}=0.04$. The approximate probability that the proportion lies between $0.12$ and $0.28$ is the normal area between standardized endpoints $-2$ and $2$, about $0.9545$. The actual sample proportion lives on the grid $0,0.01,\ldots,1$; a binomial calculation and optional continuity correction can quantify the finite-sample difference.

Increasing the number of simulated repetitions smooths the laboratory’s displayed sampling histogram. Increasing the sample size changes the underlying sampling distribution itself. These are different controls. Ten thousand simulated samples of size one do not turn the sample-size-one distribution into a normal one.

Additional practice

1. For variance 9 and sample size 36, find the standard error of a mean.
2. State what Chebyshev gives for error at least 1 in this example.
3. Why does a Cauchy population not satisfy the finite-variance CLT stated in this chapter?

Worked additional solutions

1. $3/6=0.5$.
2. At most $9/(36\cdot1^2)=0.25$. The bound is valid under its assumptions but can be loose.
3. It has no finite variance or finite ordinary mean, so the theorem’s hypotheses fail. More data alone do not supply missing moments.