A Regression for Yes and No

Suppose an outcome can only be yes or no. A straight-line regression might predict a probability of 1.4, or −.2. Neither is a probability.

Logistic regression avoids this particular embarrassment by modeling the log odds rather than probability directly.

The link from a line to a probability

Let $p(x)$ be the modeled probability of an event given predictors. Logistic regression uses

$$\log\left(\frac{p(x)}{1-p(x)}\right)=\beta_0+\beta_1x.$$

The inverse of the logit is the logistic function:

$$p(x)=\frac{1}{1+e^{-(\beta_0+\beta_1x)}}.$$

Whatever real number appears on the right-hand side of the linear predictor, the resulting probability lies between zero and one.

Odds are not probability

Odds of 1:4 correspond to a probability of $1/(1+4)=.20$.

A logistic coefficient $\beta_1$ describes a one-unit change in log odds holding the other included predictors fixed. Exponentiating gives an odds ratio: $e^{\beta_1}$.

Suppose $\beta_0=-2$ and $\beta_1=.5$. At $x=2$, the log odds are $-1$, so

$$p(2)=\frac{1}{1+e^1}\approx .269.$$

Increasing x by one multiplies fitted odds by $e^{.5}\approx1.649$ in this model. It does not multiply probability by 1.649; odds ratios and risk ratios are not the same.

A genuine R fit

Using a synthetic dataset of binary outcomes:

x <- c(0,0,1,1,2,2,3,3,4,4)
y <- c(0,0,0,1,0,1,1,0,1,1)
fit <- glm(y ~ x, family=binomial())
summary(fit)
predict(fit, newdata=data.frame(x=2), type="response")

Real models may require diagnostics, more predictors, assessment of dependence, and consideration of rare outcomes or separation.

Interpretation is conditional, not automatically causal

A predictor can be associated with the chance of an event without causing it. Study design and confounding remain central.

When communicating results, consider fitted probabilities or risk differences alongside odds ratios, and specify the population and predictors held fixed.

Check your understanding

1. Convert odds of 1:3 into probability.
2. What does $e^{\beta_1}$ represent in a logistic regression?
3. Does an odds ratio of 2 necessarily mean the event is twice as probable?

Worked responses

1. $1/(1+3)=.25$.

2. The model’s multiplicative change in odds per one-unit predictor increase, holding other terms fixed.

3. No. Odds and probability are different scales.

Fitting requires a likelihood, not squared residuals

For independent binary responses $y_i$ and modeled probabilities $p_i$, the Bernoulli likelihood is $\prod_i p_i^{y_i}(1-p_i)^{1-y_i}$. Its log is

$$\ell(\beta)=\sum_i\left[y_i\eta_i-\log(1+e^{\eta_i})\right],\qquad \eta_i=\beta_0+\beta_1x_i.$$

Logistic regression usually estimates coefficients by maximizing this expression numerically. The outcome model and optimization criterion differ from ordinary least squares. Replacing binary outcomes by logits is not a valid shortcut, because observed zeros and ones have infinite logits.

For the hypothetical coefficients $-2$ and $0.5$, the probabilities at $x=2$ and $x=3$ are approximately $0.26894$ and $0.37754$. Their difference is about $0.10860$ while their odds ratio is $e^{0.5}\approx1.64872$. At $x=0$ and $1$, the same odds ratio corresponds to probabilities about $0.11920$ and $0.18243$, a smaller difference of $0.06322$. A constant log-odds slope does not imply a constant probability difference. Differentiating the logistic function gives $dp/dx=\beta_1p(1-p)$, which makes the baseline dependence explicit.

If a coefficient estimate is $b$ with estimated standard error $s_b$, a large-sample Wald interval for its conditional odds ratio is $[e^{b-zs_b},e^{b+zs_b}]$. For $b=0.5$, $s_b=0.2$, and $z=1.96$, it is approximately $[1.114,2.440]$. This is an asymptotic interval, not an exact small-sample guarantee. Profile-likelihood methods can behave differently.

Complete separation occurs when a predictor boundary perfectly distinguishes all observed outcomes. Coefficients can diverge while the likelihood approaches a limiting supremum. A convergence warning or huge standard error should prompt inspection of the data and model, not automatic deletion of an inconvenient observation. Also inspect calibration, influential observations, and whether the logit relationship is plausibly linear in continuous predictors.

Additional practice

1. Convert probability $0.8$ to odds and log odds.
2. If baseline probability is $0.2$ and odds double, find the new probability.
3. Why does a conditional odds ratio not automatically describe a causal intervention?

Worked additional solutions

1. Odds $4$ and log odds $\log4\approx1.38629$.
2. Baseline odds are $0.25$; doubled odds are $0.5$, giving probability $0.5/1.5=1/3$.
3. The coefficient conditions on included predictors under a model. Unmeasured confounding, selection, and the distinction between observing and intervening remain.