Beat the forecaster
Each morning you get a look at the sky. Slide to your probability of rain and press "Lock in"; the day then plays out and your Brier score for it is (p − outcome)², with rain counting as 1 and dry as 0. Lower is better. Do all ten days, then compare your total with a lazy forecaster who always says 50% and with a perfectly calibrated one who knows the true chance. Notice that saying 50% is not "safe": it is beaten by anyone who reads the sky even roughly. Notice too that the calibrated forecaster can lose a day, or even a week, and still be the one you should trust.
| day | sky | you | outcome | your score | 50% score | calibrated | true p |
|---|
What's going on
A forecast of "70% chance of rain" cannot be right or wrong on a single day. Whether it rains or not, 70% allowed for it. So how do we judge a forecaster? The Brier score charges (p − outcome)² for each day. Say 90% and it rains: you pay 0.01. Say 90% and it stays dry: you pay 0.81. Say 50% and you pay exactly 0.25 whatever happens.
That 0.25 is the trap. Always saying 50% feels cautious, but anyone who can tell a clear sky from a thunderstorm will beat it over a week. If the true chance of rain is q and you say p, your expected score is (p − q)² + q(1 − q). The second part is the weather's own randomness, which nobody can remove. The first part is your error, and it is zero only when you say the true chance. So the Brier score rewards honesty: your best strategy is to report what you really believe, no hedging toward 50% and no rounding to 0% or 100%.
A forecaster is calibrated if, on the days they say 70%, it rains about 70% of the time. The calibrated forecaster here says the true probability every day. It is the best possible strategy on average, yet in any single week the dice can go against it. That is why a weather service is judged over years, not over the week your picnic was rained out.
Sheet: What "70% chance of rain" means. Related demo: Bayes' rule, on updating a probability when new evidence arrives.