What "70% chance of rain" means
Tonight's idea
The forecast said 70% chance of rain. It stayed dry. Was the forecast wrong? No: 70% allows for three dry days in ten. One day cannot catch a forecaster out. Many days can.
First, calibration. Collect every day the forecaster said 70%. It should have rained on about 70% of them. Likewise for every level.
That is not enough. If it rains on 30% of days, someone who says 30% every morning is perfectly calibrated and perfectly useless.
Second, a score. Write rain as 1, dry as 0. With forecast \(p\), the day's Brier score is
\[(p - \text{outcome})^2.\]
Lower is better. Say 0.9 and it rains: \((0.9 - 1)^2 = 0.01\). Say 0.9 and it stays dry: \(0.81\). A perfect forecaster, 1 before every wet day and 0 before every dry one, scores 0.
Say 0.5 and you score \(0.25\) whatever happens. It looks safe. It is not: a week costs \(7 \times 0.25 = 1.75\), and anyone who can read the sky at all comes in under that.
Judge over many days: calibration at each level and a low Brier total. Confident and wrong costs most.
Watch one: a five-day week
A forecaster says 0.8, 0.2, 0.6, 0.1 and 0.9 on five mornings. It rains on days 1 and 5 only. The scores are \((0.8 - 1)^2 = 0.04\), \((0.2 - 0)^2 = 0.04\), then \(0.36\), \(0.01\) and \(0.01\). Total \(0.46\).
The lazy forecaster who says 0.5 every day totals \(5 \times 0.25 = 1.25\). The perfect forecaster totals 0. Day 3 cost the most: 60% on a day that stayed dry. That forecast was not wrong, just expensive this time. The real test is whether, over many 60% days, it rained about six times in ten.
Play ten days against the lazy forecaster and a calibrated one: mathdetours.com/demos/forecast.html.
Your turn
- Score each day. (a) Forecast 0.3, it rains. (b) Forecast 0.3, dry. (c) Forecast 1, dry. (d) Forecast 0, it rains.
- A week of forecasts: 0.9, 0.7, 0.4, 0.1, 0.2, 0.6, 0.8. It rains on days 1, 2, 5 and 7. (a) Find the total Brier score. (b) Compare with always saying 0.5. (c) Which day cost the most, and was that forecast wrong?
- Over a season a forecaster said 70% on 30 days and it rained on 21 of them. She said 20% on 50 days and it rained on 15. (a) Is she calibrated at each level? (b) Find her total Brier score on the fifty 20% days. (c) What would the total have been if she had said 30% on those days instead?
- In a certain town the true chance of rain is 70% every day. In a typical ten days, seven are wet and three dry. Find the ten-day total for a forecaster who every day says (a) 0.5, (b) 0.7, (c) 0.9, (d) 1. Which wins, and why does the honest forecaster beat the confident one?
Stretch: why honesty wins
On a day with a true 70% chance of rain, a forecast \(p\) scores, on average, \(0.7(1 - p)^2 + 0.3p^2\). Expand and simplify this (see Expand and factor). Then check that it equals \((p - 0.7)^2 + 0.21\). Why does that make 0.7 the best possible forecast, and what is the 0.21?
Answers
- (a) \((0.3 - 1)^2 = 0.49\). (b) \(0.3^2 = 0.09\). (c) \(1\). (d) \(1\). Being certain and wrong costs the maximum.
- (a) \(0.01 + 0.09 + 0.16 + 0.01 + 0.64 + 0.36 + 0.04 = 1.31\). (b) Always 0.5 totals \(7 \times 0.25 = 1.75\), so this forecaster is better. (c) Day 5: 20% on a wet day, costing 0.64. Not wrong on its own; 20% allows for rain one day in five. It would be a problem only if 20% days kept turning out wet.
- (a) At 70%: \(21/30 = 70\%\), calibrated. At 20%: \(15/50 = 30\%\), not calibrated; it rained more often than she said. (b) Wet days \(15 \times 0.64 = 9.6\), dry days \(35 \times 0.04 = 1.4\), total \(11.0\). (c) \(15 \times 0.49 + 35 \times 0.09 = 7.35 + 3.15 = 10.5\). Saying what actually happens scores better.
- (a) \(10 \times 0.25 = 2.5\). (b) \(7 \times 0.09 + 3 \times 0.49 = 0.63 + 1.47 = 2.1\). (c) \(7 \times 0.01 + 3 \times 0.81 = 2.5\). (d) \(7 \times 0 + 3 \times 1 = 3\). The honest 0.7 wins. Saying 0.9 or 1 gains little on the seven wet days and pays heavily on the three dry ones, because the penalty grows with the square of the error.
Stretch. \(0.7(1 - 2p + p^2) + 0.3p^2 = 0.7 - 1.4p + p^2\). And \((p - 0.7)^2 + 0.21 = p^2 - 1.4p + 0.49 + 0.21\), the same. A square is never negative and is zero only at \(p = 0.7\), so 0.7 gives the lowest possible average. The leftover \(0.21 = 0.7 \times 0.3\) is the weather's own randomness, which no forecaster can remove.