The test that cries wolf
One thousand people take a test. Each dot is one of them, coloured by what the test said and whether it was right. You test positive. How worried should you be? Slide the three numbers and count the dots before you trust your gut. Prevalence is how common the disease is. Sensitivity is how often the test catches a sick person. Specificity is how often it clears a healthy one.
What's going on
Sensitivity is the chance the test says yes when you are sick. Specificity is the chance it says no when you are healthy. Both can be 99% and the test can still be wrong most of the time it says yes, because the false alarms come from the huge healthy group and the true alarms come from the tiny sick group. One percent of 990 healthy people is about 10 false alarms, which is as many as the 10 sick people the test catches. Half the positives are healthy.
Bayes' theorem is just this count written as a formula. The chance you are sick given a positive result is
P(sick | +) = prevalence × sensitivityprevalence × sensitivity + (1 − prevalence) × (1 − specificity)
The top is the true positives; the bottom is all the positives. Doctors call the answer the positive predictive value, and it depends on prevalence as much as on the test. The same test is reliable in a clinic full of people who already have symptoms (high prevalence) and misleading as a mass screen of healthy people (low prevalence). That is why a positive screening result is followed by a second, different test rather than by a diagnosis.
Sheets: Bayes' theorem and What a positive test really means.