Two Samples, Two Sources of Uncertainty

There are two ways to ask whether a difference is meaningful. We can ask whether two numbers are different. Or we can ask how plausible a difference of that magnitude would be under an appropriate model of uncertainty.

The first question is usually trivial. The second requires statistics.

Imagine comparing the results of two independent groups. One group has five observations, the other eight. Their sample means differ. Before celebrating, we ought to ask whether those means were estimated with comparable precision.

A pair of entirely fictional groups

Consider the following hypothetical scores:

• Group A: $8,9,10,11,12$
• Group B: $3,4,5,6,7,8,9,18$

A has a mean of 10; B has a mean of 7.5. The observed mean difference is 2.5 points.

But look at the observations. Group B is more variable, particularly because of its unusually large value of 18.

The sample variances are $s_A^2=2.5$ and $s_B^2=22$. Those values matter because the uncertainty associated with an estimated group mean depends on both sample size and variability.

Independent samples: uncertainty comes from both groups

For independent observations sampled appropriately from two populations, the estimated standard error of a difference between means is

$$\widehat{\mathrm{SE}}(\bar X_A-\bar X_B) =\sqrt{\frac{s_A^2}{n_A}+\frac{s_B^2}{n_B}}.$$

In our fictional example this is

$$\sqrt{\frac{2.5}{5}+\frac{22}{8}} =\sqrt{3.25}\approx1.803.$$

So our observed difference of 2.5 is only about $2.5/1.803\approx1.39$ estimated standard errors from zero.

This calculation does not establish whether the groups were selected appropriately. It simply describes uncertainty under a model.

Welch’s t test

A common procedure for independent means is Welch’s two-sample t test. Unlike the traditional pooled-variance Student t procedure, it does not assume the two populations have the same variance.

Its test statistic for a hypothesized zero mean difference is

$$t=\frac{\bar X_A-\bar X_B}{\sqrt{s_A^2/n_A+s_B^2/n_B}}.$$

Welch’s method uses an approximate degrees-of-freedom calculation reflecting the two sample variances and group sizes.

In R, the default independent two-sample call is:

A <- c(8, 9, 10, 11, 12)
B <- c(3, 4, 5, 6, 7, 8, 9, 18)
t.test(A, B, paired = FALSE, var.equal = FALSE)

The output includes an estimated difference, t statistic, degrees of freedom, confidence interval, and p-value. Read all of them rather than treating the p-value as the research finding.

This procedure is not appropriate for every dataset. Tiny, heavily skewed samples or extreme observations can undermine the approximation, and dependence between observations changes the variance structure.

Wait—were the observations paired?

If we measured the same eight people before and after an intervention, those are paired observations. The useful quantities are the eight within-person differences, not two independent samples.

A paired t procedure is mathematically a one-sample t procedure applied to the paired differences. Pairing can remove between-person variability, but only when the pairing genuinely describes the study design.

Do not treat two measurements on the same people as independent simply because the software accepts two columns.

A statistical test is not experimental design

Even if a two-sample test reports a very small p-value, that does not prove the group difference was caused by a treatment. The groups might differ for other reasons.

For randomized experiments, assignment helps causal interpretation; for observational comparisons, confounding remains a concern.

Exercise

1. Find the mean difference of the fictional groups above.
2. Recalculate its estimated standard error using $s_A^2=2.5$, $n_A=5$, $s_B^2=22$, and $n_B=8$.
3. Why would a paired test be inappropriate if Group A and Group B are entirely different people?

Suggested responses

1. $10-7.5=2.5$.

2. $\sqrt{2.5/5+22/8}=\sqrt{3.25}\approx1.803$.

3. There are no meaningful within-person pairs. Pairing depends on study design, not a convenient arrangement of rows.

Read further

See the official R documentation for t.test for the distinction between paired and independent tests and Welch’s default variance treatment. The ASA GAISE teaching recommendations stress inference in the context of study design.

Completing the Welch calculation

A test name is not yet a reproducible analysis. With $u=s_A^2/n_A$ and $v=s_B^2/n_B$, the Welch–Satterthwaite degrees of freedom are

$$\nu=\frac{(u+v)^2}{u^2/(n_A-1)+v^2/(n_B-1)}.$$

Here $u=0.5$, $v=2.75$, and $\nu=9.2421875$. Fractional degrees of freedom are intentional: this is an approximation matching the uncertainty in the estimated variance, not a count of people. With $SE=\sqrt{3.25}$, the statistic is $t\approx1.38675$. The two-sided p-value is approximately $0.19805$, obtained from a t distribution with that degrees-of-freedom value. Under the model and a prespecified 5% threshold we would not reject equal means. This does not establish equality.

The approximate 95% interval for $\mu_A-\mu_B$ is

$$2.5\pm t_{.975,\nu}\sqrt{3.25} \approx2.5\pm2.253153(1.802776) =[-1.56193,6.56193].$$

The interval includes both a modest negative difference and a substantial positive one on the score scale. “No significant difference” suppresses that information. State the estimate and interval before the test decision, and explain how selection and outliers could limit the result. The t approximation does not guarantee 95% actual coverage for every pair of nonnormal populations.

The variance formula follows from independence: $\operatorname{Var}(\bar X_A-\bar X_B)=\sigma_A^2/n_A+\sigma_B^2/n_B$. With correlated group means an additional $-2\operatorname{Cov}(\bar X_A,\bar X_B)$ term appears. This is why pairing is a design fact with mathematical consequences. Equal sample sizes do not create independence, and unequal sizes do not destroy it.

Further problems

1. Two independent groups have means 12 and 9, variances 16 and 9, and sample sizes 16 and 9. Calculate the estimated difference and standard error.
2. If both sample sizes are multiplied by four with the same observed variances, how does that standard error change?
3. Explain why the interval above does not imply a 95% probability that this fixed population difference lies inside these realized endpoints.
4. What changes if the observations are before-and-after scores on the same people?

Worked further solutions

1. Difference $3$; estimated standard error $\sqrt{16/16+9/9}=\sqrt2$.
2. Each variance contribution is quartered, so the standard error is halved. This assumes the same independence structure and does not address bias.
3. In the frequentist model the parameter is fixed. The repeated-sampling interval procedure carries the approximate coverage statement; these observed endpoints no longer vary.
4. Form within-person differences and analyze their mean. Their variance incorporates the covariance between before and after; applying an independent-groups formula discards that information.