Hypothesis Tests and P-Values

Few numbers in science are more confidently misunderstood than the p-value. A paper reports $p=.03$; somebody announces that there is only a 3 percent chance the finding happened by chance, or a 97 percent chance the researcher’s hypothesis is correct.

Neither statement follows.

To understand what a p-value measures, we need a model, a question, and a way of deciding what results count as unusually far from the model’s expectation.

Begin with a claim

Suppose we suspect a coin might be unfair. We plan to toss it twenty times.

Our null hypothesis is

$$H_0:p=.5,$$

where $p$ is the coin’s probability of heads on independent tosses.

A two-sided alternative says

$$H_A:p\neq.5.$$

We specify a test statistic: $X$, the number of heads in twenty independent tosses.

Under the null hypothesis, $X$ has a binomial distribution with $n=20$ and $p=.5$.

What counts as extreme?

Suppose we observe 18 heads. The null model expects an average of 10. Counts of 18, 19, and 20 are unusually high; counts of 2, 1, and 0 are equally far from the null expectation in the opposite direction.

A two-sided exact test sums their null probabilities:

$$P(X\le2\text{ or }X\ge18\mid H_0) =\frac{2(1+20+190)}{2^{20}} \approx0.00040245.$$

That is the two-sided exact binomial p-value for 18 heads in 20 tosses, using this pre-specified measure of extremeness.

Under the fair-coin model and the planned test, results at least this extreme would be rare.

But this is not $P(H_0\mid\text{data})$. The null hypothesis was an assumption inside the probability calculation, not a random event to which we assigned a posterior probability.

Significance is not effect size

If a p-value is below a threshold such as $.05$, a conventional decision rule calls the result “statistically significant.” That threshold is a choice about evidence and error rates, not a universal boundary between true and false.

With a huge sample, an extremely small difference can yield a small p-value. With a tiny sample, a large difference might not cross the threshold. Statistical significance alone does not tell us whether an effect is important, consequential, or even measured without bias.

A sensible report includes context, estimated effects, uncertainty, the design, the assumptions, and what was decided before the data were analyzed.

Manipulate the evidence

The laboratory below shows exact probabilities under the fair-coin null. Change the number of tosses and the observed number of heads.

The gold bars represent outcomes at least as far from $n/2$ as the observation; their probabilities sum to the displayed p-value.

Try 14 heads out of 20, then 18. The observation is farther from the null expectation in the second case, so the two-sided p-value is smaller.

Data snooping changes the rules

Suppose you repeatedly inspect results and stop collecting observations the instant the p-value falls below $.05$. The null distribution of your entire procedure is no longer the one assumed by a single pre-planned fixed-sample test.

Likewise, trying dozens of outcomes and only reporting the smallest p-value can make random variation look persuasive.

These problems have names—optional stopping and multiple testing—and solutions that require choosing or adjusting the analysis procedure appropriately.

Exercise

1. Interpret $p=.04$ without assigning a probability to the truth of the null hypothesis.
2. For a fair coin tossed ten times, is getting exactly five heads the strongest evidence against fairness?
3. Why should a researcher distinguish a pre-specified analysis from a result selected after examining many alternatives?

Worked answers

1. Under the null model and the chosen test, a result at least as extreme as the observed one has probability .04.

2. No. Five is exactly the null expected count and produces a two-sided p-value of 1 under the distance-from-center definition.

3. Selection and repeated examination can change false-positive rates and invalidate the nominal single-test calibration unless addressed in the method.

Read further

See OpenStax: Hypothesis Testing with One Sample and the ASA GAISE recommendations.

The tail criterion is part of the test

Under a binomial null $X\sim\mathrm{Binomial}(n,p_0)$, an upper-tail p-value is $P_{p_0}(X\geq x_{obs})$. A lower-tail value uses the opposite direction. A two-sided discrete test must specify what counts as at least as extreme. One common exact convention sums the null probabilities of all outcomes whose mass is no greater than that of the observation. It need not equal twice the smaller one-sided tail when the null distribution is asymmetric. The site’s fair-coin example is symmetric, which conceals this potential difference.

For $n=10$, $p_0=0.5$, and eight observed successes, the upper-tail probability is $(45+10+1)/1024=56/1024=0.0546875$. By symmetry, the matching two-sided tail probability is $112/1024=0.109375$. Choosing the direction after seeing eight successes and presenting the one-sided value as prespecified changes the procedure’s false-positive behavior.

A valid p-value is calibrated so that under its null $P(p\leq\alpha)\leq\alpha$; discreteness can make the inequality strict. It is not generally uniform for a discrete exact test. Optional stopping, outcome searching, and adaptive exclusions can destroy the claimed calibration if not accounted for. Report the tested model and the analysis choices that produced the statistic.

Further problems

1. For ten fair tosses and ten heads, find the one-sided and symmetric two-sided p-values.
2. Why does failing to reject not establish equivalence?
3. Why is an effect estimate with an uncertainty interval needed even after a small p-value?

Worked further solutions

1. $1/1024$ and $2/1024$, respectively.
2. Low precision can make both equality and large differences compatible with the data. Equivalence requires a prespecified practically negligible range and a suitable test.
3. A p-value does not report effect magnitude, direction on the scientific scale, or the plausible range under an estimation model.