Building a Line with Least Squares

Suppose you want to know whether a person’s hours of practice are related to how well they perform some task. You collect paired measurements and plot them.

A straight line looks plausible. Now what?

We need to decide which straight line best summarizes the relationship—and precisely what we mean by “best.”

The linear model

The familiar simple linear regression equation is

$$\widehat y=b_0+b_1x.$$

Here $b_0$ is the intercept, the predicted value when $x=0$. The slope $b_1$ is the predicted change in $y$ associated with a one-unit increase in $x$.

If the fitted line is $\widehat y=12+3x$, moving from $x=4$ to $x=5$ changes the predicted $y$ from 24 to 27.

That does not prove that changing $x$ by one unit will cause $y$ to increase by three. Regression can describe observational associations without identifying causal effects.

Also, interpreting the intercept at $x=0$ may be silly if the observed data only include $x$ values between 100 and 200.

What counts as a good line?

A residual is the difference between what we observed and what the line predicted:

$$e_i=y_i-\widehat y_i.$$

We could try to minimize the sum of residuals, but positive and negative residuals can cancel. Instead, ordinary least squares chooses the line minimizing the sum of their squares:

$$\operatorname{SSE}=\sum_{i=1}^{n}(y_i-b_0-b_1x_i)^2.$$

For data in which the $x$ values are not all identical, the least-squares slope has the convenient formula

$$b_1=\frac{\sum_i(x_i-\bar x)(y_i-\bar y)} {\sum_i(x_i-\bar x)^2}, \qquad b_0=\bar y-b_1\bar x.$$

You needn’t calculate these by hand fifty times to understand the result. Instead, understand what the optimization rewards and penalizes.

Squaring deviations makes large errors particularly expensive. That can be a good property—but it also gives extreme observations considerable influence.

A fitted line isn’t a law of nature

Fitting a line requires relatively few mathematical assumptions. Inference about the slope, however, requires more care: how were observations collected, could outcomes be dependent, does variability change across predictor values, and is a linear mean function appropriate?

For descriptive prediction, we still need to check that the line doesn’t systematically miss an obvious curve. We should also be cautious about extrapolating beyond the range where observations were collected.

What about correlation?

For simple regression with an intercept, the estimated slope and Pearson correlation have the same sign. They are related by

$$b_1=r\frac{s_y}{s_x},$$

provided both sample standard deviations are positive.

But slope and correlation are not interchangeable. Correlation has no units; slope has units of $y$ per unit of $x$. A relationship can be strongly linear while its slope is numerically very small, merely because the variables use different scales.

Exercise

1. For $(x,y)=(1,3),(2,5),(3,7)$, find the fitted line.
2. Why does ordinary least squares square residuals?
3. Is the intercept always scientifically meaningful?

Worked responses

1. Every point falls on $\widehat y=1+2x$, so the sum of squared residuals is zero.

2. Squaring avoids positive and negative cancellation and strongly penalizes larger errors. It also yields a mathematically convenient optimization problem.

3. No. The value $x=0$ might not be possible or might lie far outside the observed predictor range.

Read further

OpenStax: Linear Regression and Correlation introduces the same core ideas and applied questions.

Derive the normal equations

Minimize $S(b_0,b_1)=\sum_i[y_i-b_0-b_1x_i]^2$. Setting its partial derivatives to zero gives $\sum_i e_i=0$ and $\sum_i x_ie_i=0$. The first implies $b_0=\bar y-b_1\bar x$. Substitution into the second gives

$$b_1=\frac{S_{xy}}{S_{xx}},\qquad b_0=\bar y-b_1\bar x,$$

provided $S_{xx}>0$. The quadratic objective is minimized by these coefficients; if all predictors coincide, an intercept and slope cannot be separately identified.

For pairs $[1,2,3,4,5]$ and responses $[2,4,5,4,5]$, centered predictor squares sum to 10 and cross-products to 6. The fitted line is $2.2+0.6x$, passes through $(3,4)$, and has residuals $[-0.8,0.6,1,-0.6,-0.2]$. Their sum is zero and residual sum of squares is 2.4. The zero-sum property follows from the fitted intercept, not from a belief that individual errors cancel exactly in nature.

Least-squares fitting does not itself require normal errors. Normality, independence, and variance conditions enter particular inferential claims and prediction procedures. A deterministic algorithm can always return a line when the design is identified, even for data that make a straight-line scientific model inappropriate.

Further problems

1. Fit a least-squares line to $(0,1),(1,3),(2,5)$.
2. What changes if 10 is added to every response? If the predictor is converted from metres to centimetres?
3. Why does the fitted line with an intercept pass through the observed mean pair?

Worked further solutions

1. $\hat y=1+2x$, with zero residual sum of squares.
2. The intercept increases by 10 and slope stays fixed. Multiplying predictor values by 100 divides the numerical slope by 100, preserving fitted values.
3. The normal equation gives $b_0=\bar y-b_1\bar x$, so $b_0+b_1\bar x=\bar y$.