Show Your Work, Including the Uncomfortable Parts

A brilliant analysis that nobody can reproduce may be a brilliant story. But it is a terrible scientific artifact.

Reproducibility is not the claim that the result is correct. It is the more modest—and testable—claim that a reader can obtain the stated result from the same data and instructions.

Start with an empirical question

Write the question before the code. Define the population, unit of observation, outcome, relevant predictors, comparison, and the kind of conclusion the design supports.

“Does temperature cause ozone?” requires a causal study of interventions or strong causal identification assumptions.

“How were recorded ozone values associated with daily temperature in this historical dataset?” is an observational association question we can answer descriptively with properly documented data.

Preserve provenance

Record where the data originated, what dates they cover, what variables represent, which units they use, and whether anyone has permission to redistribute them.

Never quietly treat simulated numbers as observations from the world. Never turn a missing value into zero without disclosing and justifying the change.

For the R-distributed airquality field notebook, the original observations belong to a documented historical source and are distributed as part of R. We use them without relabeling them as our own research.

A minimal project structure

A transparent statistical project might include:

research-question.md
data-provenance.md
analysis.R
results.md
figures/
limitations.md

Every result in the final report should trace to a recorded dataset and reproducible analysis step. Specify software versions and random seeds when simulation is involved.

Report the quantities behind the headline

Include sample sizes, missing counts, relevant descriptive statistics, estimates with units, confidence intervals where appropriate, and the assumptions supporting inference.

A tiny p-value is not a substitute for an effect size. A high $R^2$ is not a prediction guarantee. An observational regression coefficient is not automatically a causal effect.

If the analysis explored many models before highlighting one, disclose the search. Exploration is legitimate; presenting it as a single preplanned confirmatory test is misleading.

The reader should be able to disagree

A good report makes limitations visible: measurement flaws, confounding, missing data, multiple comparisons, sampling-frame limits, unstable models, and the distinction between statistical and practical importance.

A source list with stable identifiers—preferably DOIs and convenient BibTeX—helps future scholars retrace your claims.

Check your understanding

1. What does reproducibility establish, and what does it not establish?
2. Why should an observational model’s conclusion avoid unsupported causal language?
3. Which items should accompany a submitted analysis script?

Worked responses

1. It allows the computation to be repeated. It does not prove assumptions, data quality, or scientific interpretation are valid.

2. The design may not isolate interventions from confounders or selection mechanisms.

3. Data source, dictionary, transformations, exclusions, software, versions, random seeds, and a clear question and interpretation.

Reproducibility is an executable chain of evidence

A useful analysis script begins with the documented source, validates the schema, applies explicit transformations, constructs the analysis population, computes results, and creates the reported tables. If a number was copied by hand from an exploratory console session, later rerunning a plotting script does not reproduce the full analysis. Record software versions, package requirements, random seeds for randomized algorithms, and acquisition dates for mutable sources.

A seed controls pseudorandom sequences for a specified implementation; it does not guarantee identical output across all software versions or make a statistical procedure scientifically valid. Record deterministic inputs and the algorithm as well. Keep exploratory work, prespecified confirmatory procedures, and later sensitivity analyses distinguishable in the report without concealing any of them.

A compact results paragraph identifies the target, design, sample and exclusions, estimate with units, uncertainty procedure, and limitations. For example, a hypothetical independent-sample mean difference of 2.5 points with a Welch interval $[-1.56,6.56]$ warrants reporting considerable uncertainty, not announcing “the groups are the same.” A reproducible script enables readers to verify arithmetic; a transparent argument enables them to challenge assumptions.

Further practice

1. List the minimum information needed to reproduce a simulation-based p-value.
2. Why should raw data and derived data be kept separately?
3. What changes when a source updates its historical records after the report is written?

Worked further responses

1. Data or a permitted access method, null model, statistic and extremeness rule, simulation algorithm, repetition count, seed, software versions, and handling of ties or corrections.
2. Separation preserves original evidence and makes transformations auditable and repeatable.
3. A fresh download may no longer reproduce the old result. Record version or retrieval date, preserve a permitted snapshot or checksum, and describe changes transparently.