Before the formula, design the study.
Statistics begins with a question, a collection of observations, and the assumptions connecting one to the other. This guided study-design workshop asks you to make those choices yourself.
1. Frame a question worth answering
A fictional university wants to know whether offering an optional statistics preparation workshop improves student success. An administrator says, “The students who attended scored higher. The workshop works.”
That sentence contains at least three distinct empirical questions:
- Description: Among the students we measured, how did scores differ between attendees and nonattendees?
- Causation: How would the same eligible students have performed if they had received the workshop rather than not received it?
- Generalization: Do conclusions about participating volunteers apply to the university's full student population?
The first can be answered by observation. The second requires a defensible causal identification strategy. The third requires attention to recruitment and representativeness. The answers need not agree.
Suggested response: what would we actually measure?
The observational unit might be a student; the outcome could be a pre-specified assessment score, pass/fail indicator, or change from baseline. We also need the treatment definition (offered a workshop versus actually attending), which cohort is eligible, and when outcomes will be measured. Merely recording enrollment in the optional workshop creates an observational exposure, not a randomized intervention.
2. Investigate the two selection decisions
First decide who enters the study. Then decide who receives the intervention. Treat these as separate experimental design choices.
Who enters the study? Who receives the intervention?
Imagine a fictional population of 1,000 people, split equally into two preparation groups. Suppose treatment would improve an outcome for each group, but by different amounts. These are assumed probabilities in a teaching model—not observations, a real trial, or simulated survey data.
| Group | Population | Without treatment | With treatment | Treatment effect |
|---|---|---|---|---|
| A · More prior preparation | 500 | 65% | 70% | +5 percentage points |
| B · Less prior preparation | 500 | 25% | 50% | +25 percentage points |
The question: how much does the treatment help in the full population? And will the study's observed difference tell us?
Before changing anything, predict whether choosing a representative sample and randomly assigning treatment are the same operation. What happens when volunteers are unusually well prepared?
The selected people are not a population-representative mix. Random assignment nevertheless preserves their expected treatment contrast.
See the numerical audit trail and group counts
| Group | Expected participants | Assigned treatment | Assigned control |
|---|---|---|---|
| A | 450 | 225 | 225 |
| B | 100 | 50 | 50 |
| Total | 550 | 275 | 275 |
Expected treated outcome rate: 66.4%. Expected control outcome rate: 57.7%. Difference: 8.6%.
In a real randomized study, the realized groups and outcomes vary by chance. This teaching model computes expected counts and probabilities, so there are no confidence intervals or sampling uncertainty. Its “effect” is defined by assumed potential-outcome probabilities; do not use these invented numbers as evidence about real educational interventions.
3. Work through four deliberate comparisons
- Default: Group A participates at 90% and B at 20%, with 50–50 random assignment within each group. Predict the difference between the population effect, the participant-group effect and the expected comparison of the two treatment arms.
- Participation changes: Set both participation rates to 50%. Does the underlying population treatment effect change? Does the expected participant-group effect now agree with it? Why?
- Who chooses treatment? Restore participation to 90% and 20%. Switch assignment to “Participants choose.” Compare the raw treated–control difference with the actual effect among study participants.
- The empty study: Set participation to zero for both groups. Why is the study contrast “undefined” rather than “zero”?
Show worked explanations after attempting the comparisons
1. The population effect is +15 percentage points, but the selected group is disproportionately drawn from A, for whom the hypothetical benefit is smaller. The expected causal effect among volunteers is approximately +8.6 points. Random assignment within the participant group yields the same expected treated-minus-control contrast of +8.6 points.
2. When both groups join at equal rates, the participant composition matches the population composition in expectation. The study-group effect becomes +15.0 points. This does not claim that every actual finite random sample is perfectly representative.
3. When 90% of A but only 10% of B elect treatment, the raw expected difference is about +31.2 points, far larger than the causal participant-group effect of +8.6. The two arms have different baseline preparation levels. The difference combines effects and preexisting selection. Observational adjustment or a stronger design would be required to isolate a causal quantity.
4. With no participants, there are no outcome rates to compare. A missing denominator produces an undefined study quantity; treating it as zero confuses absence of evidence with evidence of absence.
4. Real data are messier than our teaching model
Our numerical experiment is hypothetical. Real observations rarely provide both potential outcomes for the same person. As a contrast, look at R's documented 1973 New York air-quality dataset. It contains daily environmental measurements rather than treatment assignments. The missing observations, time structure and potential confounding should make us cautious about a causal story derived from a simple regression.
In your research memo, describe the difference between a dataset that records an association and an experiment designed to answer an intervention question. It is not necessary to run a regression yet; the point this week is to understand what the data could support.
5. Diagnose eight flawed claims
Before drafting the research memo, test whether you can distinguish sampling from assignment and see exactly which conclusions the study does—and does not—support. Try to explain each answer before revealing the feedback.
Could you spot a faulty study?
Eight short design decisions. Make your choices before checking the explanations. Nothing is sent to a server or saved.
6. Submit a defensible study-design memo
Complete the Week 1 assignment in two pages. State a question, population, unit of observation, measurement and assignment mechanism; include a simple diagram of recruitment and assignment. Identify one threat to each of measurement quality, causal identification and population generalization.
What earns a strong evaluation? Explicit definitions; a feasible recruitment procedure; a plausible treatment/comparison design where appropriate; ethical considerations; and a conclusion that does not exceed the evidence.
Instructor guide and discussion prompts
Ask the class whether random assignment repairs an unrepresentative sample (it does not). Ask whether a representative sample compensates for self-selection into treatment (it does not). Emphasize that the numeric lab displays expected counts, not Monte Carlo realizations. Have students explain why treatment effects that differ by subgroup cause transportability problems. For the memo, distinguish the effect of being offered a workshop from the effect of attendance. No method recovers unobserved potential outcomes without assumptions.
For accessibility, have learners describe the two design decisions verbally, use the numerical audit table, and navigate all controls by keyboard. Invite multiple defensible answers, provided students can state the assumptions and limitations.
Reading and next step
1. Why Statistics? · 2. Data, Variables, and Measurement · 3. What Sort of Study Are We Actually Doing?