Bootstrapping the Sampling Distribution

Suppose we want to estimate the average waiting time at a particular service desk. We have twelve observations, not twelve thousand. And one person had to wait much longer than everybody else.

We can calculate the mean. But how uncertain should we be about it?

Earlier we learned that sample statistics change from sample to sample. Ideally, we would repeatedly collect new samples from the entire population and study how our estimates vary. That isn’t usually practical.

The bootstrap offers a clever approximation: repeatedly create new samples from the data we already have.

A sample becomes our temporary population

Suppose our observed waiting times in minutes are:

$$4,\;5,\;6,\;6,\;7,\;8,\;9,\;9,\;11,\;13,\;18,\;39.$$

Our sample contains twelve observations. A bootstrap replicate draws twelve values with replacement from those twelve, selecting each draw independently and giving each original observation equal probability.

With replacement means a value may appear multiple times in one replicate, while another value may not appear at all.

For each replicate, calculate the mean. Repeat perhaps 1,000 times. The resulting distribution of 1,000 means is the bootstrap distribution.

It gives us information about how the mean might vary under repeated samples from the empirical distribution, the distribution that puts probability $1/12$ on each observed data point.

Why replacement matters

If we sampled twelve distinct positions without replacement from twelve original observations, every replicate would contain all twelve values. The mean would never vary.

We wouldn’t have learned anything about uncertainty.

Sampling with replacement introduces variation consistent with treating our observed dataset as a provisional stand-in for the population.

A bootstrap standard error

The standard deviation of the bootstrap means estimates the standard error of our original statistic. For suitable statistics and regularity conditions, the bootstrap can approximate sampling behavior without requiring us to work out a complicated formula for that statistic.

This is not magic. The bootstrap inherits the information—and the blind spots—of the original sample.

If our observations were selected badly, resampling the same biased dataset does not repair the bias.

Percentile intervals

A simple percentile bootstrap confidence interval uses quantiles of the bootstrap distribution.

For a nominal 95 percent interval, take the 2.5th and 97.5th percentiles of the bootstrap replicate estimates:

$$[\widehat q_{0.025},\widehat q_{0.975}].$$

The lab below calculates a Monte Carlo approximation to these endpoints. Change the number of replicates and watch the endpoints fluctuate. Generating more bootstrap replicates reduces Monte Carlo noise; it does not create new information from the population.

This interval is easy to explain, but it is not universally well-calibrated. Small or skewed samples, bias, boundary parameters, dependence, and nonsmooth statistics can produce poor coverage. Studentized, bias-corrected, or otherwise tailored methods may be more appropriate.

The lesson is not that the percentile bootstrap replaces every confidence interval. It is that uncertainty can sometimes be approximated through carefully specified computational repetition.

Experiment

Look first at the original mean, then at the spread of the resampled means.

Notice that the exceptionally long wait of 39 minutes can be repeated several times—or not at all—in a bootstrap replicate. This affects the right tail of the bootstrap distribution.

Try 200 and then 3,000 replicates. The shape and endpoints will generally become more numerically stable with the larger count, although they will still vary.

Exercise

1. Why do bootstrap samples have the same size as the original sample in this standard procedure?
2. Why is sampling without replacement not useful for the mean in this example?
3. Would creating a million bootstrap replicates make a convenience sample representative?

Worked responses

1. We are approximating the distribution of the statistic at the original sample size under repeated sampling from the empirical distribution.

2. A twelve-observation sample without replacement from the same twelve observations contains exactly the same values, and therefore the same mean.

3. No. Additional replicates primarily improve the precision of the computational approximation, not the quality of the underlying evidence.

Read further

OpenStax Introductory Statistics 2e provides the conventional sampling-and-interval framework into which resampling methods fit. The ASA GAISE recommendations encourage computing to emphasize conceptual understanding.

What increasing bootstrap repetitions can and cannot do

For sample $x_1,\ldots,x_n$, the empirical distribution places probability $1/n$ at each observed record, counting repeated values with their multiplicity. A bootstrap sample draws $n$ times independently from that empirical distribution. More bootstrap repetitions reduce Monte Carlo error in summarizing this conditional distribution. They do not add original observations or make an unrepresentative sample representative.

For a mean, the empirical-distribution variance is $v_n=\sum_i(x_i-\bar x)^2/n=(n-1)s^2/n$. The exact conditional variance of a bootstrap sample mean is $v_n/n=(n-1)s^2/n^2$. Thus the bootstrap’s plug-in standard error differs slightly from $s/\sqrt n$ in a tiny sample; this finite-sample effect is not a software bug.

For the sample $[1,2,3]$, the original mean is 2 and sample variance is 1. The empirical variance is $2/3$, so the exact bootstrap mean variance is $2/9$ and its standard deviation $\sqrt2/3\approx0.471405$. The conventional estimated sampling standard error is $1/\sqrt3\approx0.57735$. Enumerating all $3^3=27$ ordered resamples can verify the conditional result without simulation noise.

Ordinary resampling of individual records assumes the observed units are suitably independent. For paired observations, resample pairs. For cluster samples, a design-respecting cluster bootstrap may be needed. Time series require dependence-aware methods. Percentile intervals also inherit estimator bias and skewness issues; an impressive number of repetitions does not solve them.

Further problems

1. Why can an ordinary bootstrap sample contain repeated observations while omitting others?
2. If only five original people were measured, how many original people have been studied after a million bootstrap repetitions?
3. Why is the ordinary bootstrap poorly suited to learning about unseen extreme tails from a tiny sample?

Worked further solutions

1. Sampling is with replacement from the empirical distribution; each draw can select any original record.
2. Five. Resampling quantifies a conditional approximation, not new recruitment.
3. The empirical distribution assigns no mass beyond observed values. It cannot invent tail information absent from the data without additional modeling assumptions.