Bayes' theorem
Tonight's idea
The probability of \(A\) given \(B\), written \(P(A \mid B)\), is the share of the \(B\)-cases that are also \(A\)-cases: \(P(A \mid B) = P(A \text{ and } B)/P(B)\). Swap the letters: \(P(B \mid A) = P(A \text{ and } B)/P(A)\). Equate the two expressions for \(P(A \text{ and } B)\) and divide by \(P(B)\):
\(\displaystyle P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}\). Bayes' theorem turns "how likely is the evidence if the hypothesis is true" into "how likely is the hypothesis now that we have the evidence".
\(P(A)\) is the prior, \(P(B \mid A)\) the likelihood, \(P(A \mid B)\) the posterior. The denominator adds up both routes to the evidence: \(P(B) = P(B\mid A)P(A) + P(B \mid \text{not } A)P(\text{not } A)\).
Natural frequencies. Imagine 10,000 people: 1% are sick, the test catches 99% of cases and flags 5% of the healthy. The tree counts both routes to a positive: \(P(\text{sick} \mid \text{positive}) = 99/594 \approx 17\%\). A "99% accurate" test, and five positives in six are false alarms: the base rate at work.
Odds form. The odds of \(A\) are \(P(A) : P(\text{not } A)\); odds \(4{:}1\) means probability \(\tfrac45\). Divide Bayes for \(A\) by Bayes for not-\(A\) and \(P(B)\) cancels: posterior odds = likelihood ratio × prior odds, where the likelihood ratio is \(P(B \mid A)/P(B \mid \text{not } A)\). Each clue multiplies the odds by its own ratio, provided the clues are independent given the hypothesis: a second clue must add something new.
Watch one: the two coins
A bag holds a fair coin and a double-headed coin. You pull one out, flip it, and get heads. Which coin is it?
Formula. Prior \(\tfrac12\). Likelihoods \(P(\text{H} \mid \text{DH}) = 1\) (DH = double-headed), \(P(\text{H} \mid \text{fair}) = \tfrac12\). \(P(\text{H}) = \tfrac12 + \tfrac14 = \tfrac34\). Posterior \(P(\text{DH} \mid \text{H}) = \tfrac12 / \tfrac34 = \tfrac23\).
Frequencies. Do it 400 times: 200 double-headers give 200 heads, 200 fair coins give about 100. Of 300 heads, 200 came from the double-header: \(\tfrac23\).
Update again. A second flip is also heads. Odds form: prior odds \(2{:}1\), likelihood ratio 2, posterior odds \(4{:}1\), so \(P(\text{DH}) = \tfrac45\).
Your turn
- A disease affects 1 person in 1,000. The test has sensitivity 99% and wrongly flags 1% of healthy people. Imagine 100,000 people: count the sick, the sick who test positive and the healthy who test positive, then find \(P(\text{sick} \mid \text{positive})\). (Demo: mathdetours.com/demos/bayes.html.)
- Prosecutor's fallacy. A DNA profile is shared by 1 person in 100,000. The culprit is one of a city's 2,000,000 people, and there is no other evidence. About how many innocent people in the city share the profile, and what is \(P(\text{guilty} \mid \text{match})\) for a matching person?
- Spam filter. 30% of your mail is spam. "free" appears in 50% of spam and 5% of real mail. A message contains "free": \(P(\text{spam})\)? It also contains "meeting" (2% of spam, 20% of real mail). Assume the two words occur independently within spam and within real mail (a simplified "naive Bayes" filter): multiply both likelihood ratios and find the new \(P(\text{spam})\).
- Monty Hall in one line. You pick door 1. The host, who knows where the car is and always opens a goat door (at random if he has two), opens door 3. Write down \(P(\text{opens 3} \mid \text{car behind } k)\) for \(k = 1, 2, 3\), and use Bayes to find \(P(\text{car behind } 2 \mid \text{opens 3})\).
Stretch: the taxi problem
A taxi was in a hit-and-run at night. In this city 85% of taxis are green and 15% blue. A witness says blue, and under night conditions witnesses get colours right 80% of the time. Most people answer 80%. Use the odds form. A second witness, whose errors are independent of the first's given the taxi's true colour, also says blue: update again. Then a photocopy of the first statement arrives: why not multiply by 4 again?
Answers
- 100 sick, 99 positive. 99,900 healthy, 999 positive. \(P(\text{sick} \mid \text{positive}) = 99/1098 \approx 9\%\). The 1% error rate acts on a huge healthy population, so false positives swamp true ones.
- Innocent matches: \(2{,}000{,}000 / 100{,}000 = 20\). The culprit is one of about 21 matching people, so \(P(\text{guilty} \mid \text{match}) \approx 1/21 \approx 5\%\), not \(99.999\%\). The "one in 100,000" is \(P(\text{match} \mid \text{innocent})\), the wrong way round.
- Spam route \(0.3 \times 0.5 = 0.15\); real route \(0.7 \times 0.05 = 0.035\); \(P(\text{spam}) = 0.15/0.185 \approx 0.81\). Add "meeting": \(0.15 \times 0.02 = 0.003\) against \(0.035 \times 0.2 = 0.007\); \(P(\text{spam}) = 0.3\). In odds: \(3{:}7\), times 10, times \(0.1\), back to \(3{:}7\). Without the independence assumption the answer is not determined: if every spam "meeting" also carries "free" and every real "free" also carries "meeting", the routes are \(0.3 \times 0.02\) against \(0.7 \times 0.05\) and \(P(\text{spam}) = 6/41 \approx 0.15\).
- \(\tfrac12\) (he could open 2 or 3), \(1\) (door 2 is off limits), \(0\). Priors \(\tfrac13\) each. \(P(\text{car } 2 \mid \text{opens 3}) = \dfrac{1 \cdot \tfrac13}{\tfrac12 \cdot \tfrac13 + 1 \cdot \tfrac13 + 0} = \tfrac23\). Switch.
Stretch. Prior odds blue:green \(= 15{:}85 = 3{:}17\). Likelihood ratio \(0.8/0.2 = 4\). Posterior odds \(12{:}17\), so \(P(\text{blue}) = 12/29 \approx 0.41\): green is still more likely. Second witness: odds \(48{:}17\), \(P(\text{blue}) = 48/65 \approx 0.74\). The photocopy is the same report again: whatever the taxi's colour, it says blue with certainty, so its likelihood ratio is 1 and the odds stay \(48{:}17\). Evidence counts once; the second witness counted because, for a taxi of either colour, his error was independent of the first's. (Kahneman and Tversky's taxi problem.)