What a positive test really means
Tonight's idea
A test for a disease is "99% accurate". You test positive. How worried should you be? Most people, doctors included, say about 99%. The real answer can be under 1%.
A test has two accuracies, not one. Sensitivity is the chance it says positive when you do have the disease: \(P(+ \mid D)\), read "the probability of a positive, given disease". Specificity is the chance it says negative when you don't: \(P(- \mid \text{no } D)\). The third number, the one the headline leaves out, is the prevalence: how common the disease is among people like you before any test.
What you want is the reverse conditional \(P(D \mid +)\), the positive predictive value. Imagine a crowd of \(N\) people and sort them into four boxes: sick or not, positive or not. The sick row holds prevalence × \(N\) people, the healthy row holds the rest.
Then \(P(D \mid +) = \dfrac{\text{true positives}}{\text{true positives} + \text{false positives}}\). When the disease is rare, the "no disease" row is enormous, and even a 1% false-positive rate applied to it produces more false positives than there are sick people. As a formula this is Bayes' theorem:
\(P(D \mid +) = \dfrac{\text{sens} \times \text{prev}}{\text{sens} \times \text{prev} + (1 - \text{spec}) \times (1 - \text{prev})}\).
Slide the three numbers at mathdetours.com/demos/bayes.html.
Watch one: 1 in 1,000, with a 99% / 99% test
Take 100,000 people. Prevalence 1 in 1,000 means 100 have the disease and 99,900 do not. Rows are the truth, columns the test result:
| Test + | Test − | Total | |
|---|---|---|---|
| Disease | 99 | 1 | 100 |
| No disease | 999 | 98,901 | 99,900 |
| Total | 1,098 | 98,902 | 100,000 |
Of the 1,098 who test positive, 99 are sick: \(P(D \mid +) = 99/1098 = 9.0\%\). A positive result leaves you with a 91% chance of being healthy. A negative result is almost perfectly reassuring: \(98{,}901/98{,}902 = 99.999\%\).
Your turn
- A test has sensitivity 90% and specificity 95%, and the disease has prevalence 2%. Fill in the table for 10,000 people and find the positive predictive value and the negative predictive value \(P(\text{no } D \mid -)\).
Test + Test − Total Disease No disease Total 10,000 - The same test is used in a specialist clinic where 30% of the patients referred turn out to have the disease. Redo the table for 10,000 patients. What is the positive predictive value now? What changed?
- A disease affects 1 person in 10,000. The test is 99% sensitive and 99% specific. Using a crowd of 1,000,000, find \(P(D \mid +)\).
- Use your answer to question 3 as the new prevalence. You test positive, then take a second test (same sensitivity and specificity, errors independent of the first's for a given patient), and it is also positive. Find \(P(D \mid ++)\). Why do doctors confirm a screening result with a second test?
Stretch: what a p-value is, with a coin
You flip a coin 20 times and get 15 heads. The p-value of this one-sided test is the chance a fair coin gives 15 or more heads: \(\dfrac{\binom{20}{15} + \cdots + \binom{20}{20}}{2^{20}}\), numerator 21,700. Compute it. Is it below the usual 0.05 cut-off? (One-sided: only too many heads count.) Now 20 researchers each flip a different fair coin 20 times, each calling 15 or more heads "significant". What is the chance at least one does?
Answers
- 200 have the disease: 180 true positives, 20 false negatives. 9,800 do not: 490 false positives, 9,310 true negatives. Column totals 670 and 9,330. PPV \(= 180/670 = 26.9\%\). NPV \(= 9310/9330 = 99.8\%\). Three in four positives are false.
- 3,000 sick: 2,700 true positives, 300 false negatives. 7,000 healthy: 350 false positives, 6,650 true negatives. PPV \(= 2700/3050 = 88.5\%\). The test did not change; the base rate did. The same positive result is strong evidence in the clinic and weak evidence in the general population.
- 100 sick: 99 true positives, 1 false negative. 999,900 healthy: 9,999 false positives, 989,901 true negatives. \(P(D \mid +) = 99/(99 + 9999) = 0.98\%\). Ninety-nine out of every hundred positives are false.
- New prevalence 0.98%. \(P(D \mid ++) = \dfrac{0.99 \times 0.0098}{0.99 \times 0.0098 + 0.01 \times 0.9902} = \dfrac{0.00970}{0.00970 + 0.00990} = 49.5\%\). Two positives take you from 1-in-10,000 to roughly a coin flip. A confirmatory test is cheap compared with treating 99 healthy people for every sick one. The independence matters: rerunning the same test on the same sample would mostly repeat the same error and add little.
Stretch. \(21700/1048576 = 0.0207\), below 0.05, so the result counts as "significant". Each researcher's false-alarm rate is the actual size of the rule, 0.0207, not the 0.05 cut-off (14 or more heads would already give 0.058): \(P(\text{at least one false alarm}) = 1 - (1 - 0.0207)^{20} \approx 0.34\). A test with false-alarm rate exactly 5% would give \(1 - 0.95^{20} \approx 0.64\). A study that searches many things for a link will find "significant" ones by chance; the same lesson as the medical test.