Not Significant Does Not Mean No Effect

A study reports $p=.21$ and announces that the treatment has no effect. This is a familiar leap—and an unjustified one.

The study may have found too little evidence to reject the null hypothesis. It has not necessarily proved the effect is zero. Perhaps it was far too small to detect a useful difference.

Two ways a decision can be wrong

Suppose we specify a significance threshold $\alpha=.05$ before collecting data.

A Type I error is rejecting the null hypothesis when that hypothesis is true. Under the correctly calibrated test, the probability of this error is controlled at the chosen threshold.

A Type II error is failing to reject the null under a specified alternative when the null is false. Denote its probability by $\beta$.

Statistical power is $1-\beta$: the probability that the planned procedure rejects the null when that particular alternative is true.

Power is not a single universal property of a dataset. It depends on the effect being considered, the test, sampling variability, sample size, significance threshold, and underlying assumptions.

What improves power?

Increasing sample size usually reduces standard errors in appropriate independent sampling models, making a fixed effect easier to distinguish from chance variation.

Using more precise measurements or a design that removes irrelevant between-person variation can also help.

Lowering the significance threshold from .05 to .01 tends to reduce false-positive rejection under the null—but usually reduces power for a fixed alternative, all else equal.

The trade-off belongs in the design discussion, not a post-hoc explanation of an inconvenient p-value.

Effect size has a scientific scale

A statistically detectable difference of .01 unit might be trivial or enormously consequential depending on the scientific context.

Report the original units and a meaningful range of consequences. Standardized effect measures such as Cohen’s $d$ can help compare scales but do not replace subject-matter interpretation.

A huge study may report a tiny p-value for a minuscule effect. A small study can fail to reject even when data are compatible with large positive and negative effects. Confidence intervals help show the range of effects still compatible with the observations under a model.

Plan rather than retrofit

Sample-size planning should identify an effect worth detecting, plausible variability, desired power, significance threshold, allocation ratio, missing data, and the analysis strategy.

Choosing the “minimum interesting effect” after seeing the result changes the question. Pretending to have planned something that was chosen afterward is worse.

Check your understanding

1. What probability does power describe?
2. Does $p=.21$ establish no difference between populations?
3. Why can a very large study find a practically negligible difference to be statistically significant?

Worked responses

1. The probability of rejecting the null for a particular specified alternative under the planned procedure.

2. No. It indicates insufficient evidence to reject at common thresholds, not proof of zero effect.

3. With enough precision, even a tiny nonzero deviation can be statistically detectable.

Read ASA GAISE recommendations for interpretation in context.

Calculate power for a specified alternative

Consider an idealized independent normal sample with known $\sigma$. Test $H_0:\mu=\mu_0$ against a two-sided alternative using $|Z|>z_{1-\alpha/2}$, where $Z=(\bar X-\mu_0)/(\sigma/\sqrt n)$. Under a true alternative $\mu_1$, $Z$ is normal with unit variance and mean $d=(\mu_1-\mu_0)\sqrt n/\sigma$. Power is

$$1-\Phi(z_{1-\alpha/2}-d)+\Phi(-z_{1-\alpha/2}-d).$$

For $d=2$ and a critical value approximately $1.96$, power is about $0.516$. The same test can have high power for a large alternative and low power for a small one; attaching one power number without an alternative is incomplete.

For planning a two-sided mean test, a useful normal approximation is $n\approx[(z_{1-\alpha/2}+z_{1-\beta})\sigma/\Delta]^2$, where $\Delta$ is the magnitude worth detecting. With $\sigma=10$, $\Delta=5$, 5% significance, and 80% desired power, this gives about $31.4$, so round up to 32 before allowing for attrition. This is approximate, assumes the known-spread model, and differs from a full t-based calculation.

Power calculated after observing an effect estimate does not add independent evidence to the corresponding p-value. Instead report an interval and examine whether it excludes scientifically meaningful effects. Demonstrating practical equivalence requires prespecified equivalence margins and a suitable procedure, not merely failing to reject an equality null.

Further problems

1. Under the planning approximation, what happens to required sample size if the effect worth detecting is halved?
2. What happens if the standard deviation doubles?
3. Explain why a 5% Type I error rate is not the probability that a particular significant conclusion is false.

Worked further solutions

1. It quadruples because sample size scales as $1/\Delta^2$.
2. It quadruples because sample size scales as $\sigma^2$.
3. The error rate conditions on the null being true and describes repeated use of the procedure. The probability of a false conclusion after selection also depends on the mix of true and false hypotheses, power, and reporting practices.