What a p-value is, and is not
Tonight's idea
A friend says her coin favours heads. Ten flips give eight heads. Suspicious. But a fair coin can do that.
Assume nothing is going on: the coin is fair. That is the null hypothesis, \(H_0\). Ask: if \(H_0\) is true, how likely is a result at least as extreme as ours? That probability is the p-value. Since the claim was about heads, "at least as extreme" means eight or more (a one-sided test). All \(2^{10} = 1024\) sequences are equally likely, so
\[p = \frac{\binom{10}{8} + \binom{10}{9} + \binom{10}{10}}{1024} = \frac{45 + 10 + 1}{1024} = \frac{56}{1024} \approx 0.055.\]
A fair coin does this about one time in eighteen. Convention calls a result significant when \(p \lt 0.05\). Ours just misses; one more head gives \(p = 11/1024 \approx 0.011\). Nothing changes at 0.05; Fisher suggested it in 1925 and it stuck.
The p-value is \(P(\text{data} \mid H_0)\), the chance of the data given a fair coin. It is not \(P(H_0 \mid \text{data})\), the chance the coin is fair given the data. Swapping them is the medical-test mistake: the second needs to know how common biased coins are. Nor does it measure how biased the coin is; a large p-value does not prove fairness.
Watch one: 60 heads in 100 flips
The exact one-sided p-value is the binomial tail
\[p = \sum_{k=60}^{100} \binom{100}{k} \Big/ 2^{100} \approx 0.0284.\]
The bell curve gets close without the sum: the count of heads has mean 50 and SD 5, so 60 is two SDs up, and about 0.023 lies beyond. (Starting the tail at 59.5, because counts are whole numbers, gives 0.029.) One-sided, this is significant. But if the claim had only been "unfair", with no direction, 40 or fewer heads would count as extreme too, and the two-sided p-value is \(2 \times 0.0284 = 0.057\). Not significant. Same data, different question, different verdict.
Your turn
- Ten flips of a coin claimed to favour heads. Find the one-sided p-value for (a) nine heads, (b) ten heads.
- Sixteen flips give twelve heads. Compute the one-sided and two-sided p-values exactly (you will need \(\binom{16}{12}\) up to \(\binom{16}{16}\)). What is the verdict at 0.05 in each case?
- In a bag of coins, one in fifty is weighted so that heads comes up with probability 0.8; the rest are fair. You draw one at random, flip it ten times, and get eight heads. (a) Show that \(P(\text{8 or more} \mid \text{weighted}) \approx 0.678\). (b) Use Bayes' theorem to find \(P(\text{weighted} \mid \text{8 or more})\). Compare with the tempting "the coin is 94.5% likely to be biased".
- Suppose the coin really does give heads with probability 0.6. You use the rule "ten flips; declare it biased if 8 or more are heads". What is the chance the rule catches the bias? What, then, does a non-significant result tell you?
Stretch: the smallest experiment
All heads in \(n\) flips has one-sided p-value \(1/2^n\). What is the smallest \(n\) for which all heads is significant at 0.05? And two-sided? Now suppose a coin's true probability of heads is 0.7. How often does the smallest one-sided experiment reach significance?
Answers
- (a) \((10 + 1)/1024 = 11/1024 \approx 0.0107\). (b) \(1/1024 \approx 0.001\).
- \(\binom{16}{12} + \cdots + \binom{16}{16} = 1820 + 560 + 120 + 16 + 1 = 2517\), and \(2^{16} = 65536\). One-sided \(p = 2517/65536 \approx 0.038\), significant. Two-sided \(p \approx 0.077\), not significant.
- (a) \(\binom{10}{8}(0.8)^8(0.2)^2 + \binom{10}{9}(0.8)^9(0.2) + (0.8)^{10} = 0.302 + 0.268 + 0.107 = 0.678\). (b) \(P(\text{weighted} \mid 8+) = \dfrac{0.02 \times 0.678}{0.02 \times 0.678 + 0.98 \times 0.0547} = \dfrac{0.01356}{0.01356 + 0.05359} \approx 0.20\). Eight heads leaves an 80% chance the coin is fair. The "94.5%" is \(1 - p\), which is the chance a fair coin gives fewer than eight heads; it says nothing about this coin until you know how many coins are weighted.
- \(P(8+ \mid 0.6) = 45(0.6)^8(0.4)^2 + 10(0.6)^9(0.4) + (0.6)^{10} = 0.121 + 0.040 + 0.006 = 0.167\). The rule misses a genuine 60/40 coin five times out of six. A non-significant result means the data are consistent with fairness, not that fairness is established. Ten flips are far too few to see a bias this size; this chance of detection is called the power of the test.
Stretch. \(1/2^4 = 0.0625\) fails and \(1/2^5 = 0.03125\) passes, so \(n = 5\). Two-sided the p-value is \(2/2^n\): \(2/32 = 0.0625\) fails, \(2/64 = 0.03125\) passes, so \(n = 6\). With a 0.7 coin, five heads in five flips happens with probability \(0.7^5 \approx 0.168\): the smallest experiment detects even a strongly biased coin only one time in six.
How evidence from many small studies is weighed, and why p-values alone are not enough, is the subject of our research page.