The line of best fit, computed
Tonight's idea
In grade 8 you drew a line of best fit by eye. Here is how to compute it. For a line \(y = a + bx\), each point has a residual: the vertical gap between the point and the line. The least-squares line is the one that makes the sum of the squared residuals as small as possible.
Let \(\bar x\) and \(\bar y\) be the means of the \(x\) and \(y\) values. The line passes through \((\bar x, \bar y)\), and its slope is \(b = \dfrac{\sum (x - \bar x)(y - \bar y)}{\sum (x - \bar x)^2}\). Then \(a = \bar y - b\bar x\).
Read the top as a vote. A point right of \(\bar x\) and above \(\bar y\), or left and below, has a positive product and pushes the slope up. The other two corners push it down. Points far from the means vote loudest. The proof comes next year.
How good is the fit? Compute \(r^2 = \dfrac{\left(\sum (x - \bar x)(y - \bar y)\right)^2}{\sum (x - \bar x)^2 \,\sum (y - \bar y)^2}\). It lies between 0 and 1: the fraction of the variation in \(y\) that the line accounts for. It hides a curve, a wild point, and whether \(x\) causes \(y\) at all.
Watch one: five points
The means are \(\bar x = 3\) and \(\bar y = 4\). Subtract them, then multiply and square.
| \(x\) | \(y\) | \(x - \bar x\) | \(y - \bar y\) | product | \((x - \bar x)^2\) | \((y - \bar y)^2\) |
|---|---|---|---|---|---|---|
| 1 | 2 | \(-2\) | \(-2\) | 4 | 4 | 4 |
| 2 | 4 | \(-1\) | 0 | 0 | 1 | 0 |
| 3 | 5 | 0 | 1 | 0 | 0 | 1 |
| 4 | 4 | 1 | 0 | 0 | 1 | 0 |
| 5 | 5 | 2 | 1 | 2 | 4 | 1 |
| Sum | 6 | 10 | 6 | |||
So \(b = \tfrac{6}{10} = 0.6\) and \(a = 4 - 0.6 \times 3 = 2.2\). The line is \(y = 2.2 + 0.6x\), and \(r^2 = \tfrac{6^2}{10 \times 6} = 0.6\): the line accounts for 60% of the variation in \(y\).
Place your own points and watch the line, the residuals and \(r\) update. Tick "mean lines" to see the line pass through the point of means: mathdetours.com/demos/regression.html.
Your turn
- Find the least-squares line for (1, 1), (2, 3), (3, 2), (4, 5), (5, 4). Use it to predict \(y\) when \(x = 6\).
- Find the least-squares line and \(r^2\) for (0, 1), (2, 2), (4, 6), (6, 7).
- The points \((-2, 4)\), \((-1, 1)\), (0, 0), (1, 1), (2, 4) all lie on \(y = x^2\). Find the least-squares line and \(r^2\). What does this value of \(r^2\) tell you, and what does it not tell you?
- The points (1, 1), (2, 2), (3, 3), (4, 4) lie on a line, so \(b = 1\) and \(r^2 = 1\). Someone adds a fifth point, (5, 0). Recompute the line and \(r^2\). Why does one point have so much power here?
Stretch: a point with no vote
Ten students have mean study time \(\bar x = 3\) hours and mean mark \(\bar y = 60\). Their least-squares line has slope 4. An eleventh student studies 3 hours and scores 93. Find the new line. Why can this student not change the slope?
Answers
- \(\bar x = 3\), \(\bar y = 3\). Products: \(4 + 0 + 0 + 2 + 2 = 8\). \(\sum (x - \bar x)^2 = 10\). So \(b = 0.8\) and \(a = 3 - 0.8 \times 3 = 0.6\): the line is \(y = 0.6 + 0.8x\). At \(x = 6\) it predicts \(y = 5.4\).
- \(\bar x = 3\), \(\bar y = 4\). The \(x\) gaps are \(-3, -1, 1, 3\) and the \(y\) gaps are \(-3, -2, 2, 3\). Products: \(9 + 2 + 2 + 9 = 22\). \(\sum (x - \bar x)^2 = 20\) and \(\sum (y - \bar y)^2 = 26\). So \(b = 1.1\), \(a = 4 - 3.3 = 0.7\), and the line is \(y = 0.7 + 1.1x\). \(r^2 = \tfrac{22^2}{20 \times 26} = \tfrac{484}{520} \approx 0.93\).
- \(\bar x = 0\), \(\bar y = 2\). Products: \((-2)(2) + (-1)(-1) + 0 + (1)(-1) + (2)(2) = 0\). So \(b = 0\), the line is the flat line \(y = 2\), and \(r^2 = 0\). That says no straight line helps predict \(y\). It does not say \(x\) and \(y\) are unrelated: here \(x\) fixes \(y\) exactly. Always look at the picture.
- Now \(\bar x = 3\) and \(\bar y = 2\). Products: \((-2)(-1) + (-1)(0) + 0 + (1)(2) + (2)(-2) = 0\). So \(b = 0\), the line is \(y = 2\), and \(r^2 = 0\). The new point is as far from \(\bar x\) as any point can be here, and far below \(\bar y\), so its vote of \(-4\) cancels all the others. Squaring makes big residuals very costly, so the line tilts a long way to shorten one.
Stretch. The new point has \(x = \bar x\), and \(\bar x\) stays at 3, so \(x - \bar x = 0\): it adds nothing to the top or the bottom of the slope formula. The slope is still 4. The new mean mark is \(\tfrac{600 + 93}{11} = 63\), so the line is \(y = 63 + 4(x - 3) = 51 + 4x\). The whole line moves up 3 marks without tilting.