Followup to Lying with Statistics V: The rules of 3, 5, 7, 9, and 10

(Note that this post is more for my own reference than anything else, but if you find it interests you, then that’s wonderful.)

In the “setup”:http://www.randomjohn.info/wordpress/2006/05/10/lying-with-statistics-v-the-very-rare-and-not-found-adverse-event/ of the rare occurrence (say, of an adverse event), the standard way of computing approximate confidence intervals doesn’t work very well. In the extreme case of the possible, but unobserved, occurrence, the standard way of computing gives a confidence interval of 0 to 0 — i.e. the occurrence is considered impossible. The apparent paradox is resolved by realizing that the standard way of computing approximate confidence intervals for a rare event is incorrect and far off the mark. For the unobserved occurrence, a better confidence interval for the rate of occurrence is between 0 and 3/_n_, where _n_ is the sample size of the study. This is called the “Rule of 3.” (More info at the link.)

Similar rules of thumb exist for the “scarcely occurring” event — i.e. one that is observed 1, 2, 3, or 4 times. You can find the rules in the following table:

|*Occurrences in study*|*Rule*|*95% Confidence Interval for estimated occurrence*|
|0|3|0 – 3/n|
|1|5|0 – 5/n|
|2|7|0 – 7/n|
|3|9|0 – 9/n|
|4|10|0 – 10/n|

(I can’t remember the reference right off the top of my head; it’s a letter in ??JAMA(Journal of the American Medical Association)??, vol 274, p. 1013(I think), 1994.)

The rule of 3 works very well for sample sizes over 30 (percent error from true 95% UCL(Upper Confidence Limit) is around 5% at around 30, and goes down fairly quickly). However, I’ve noticed that the rules of 5, 7, 9, and 10 seem quite conservative (larger interval than warranted). I started poking around at the rule of 5 and noticed that a “rule of 4.75” seems to work better. I did a little calculus similar to the one that Steve Simon did for the rule of 3, and showed that if you are going to have a “rule of k” where k/n is the 95% UCL(Upper Confidence Limit), then 4.75 works a bit better than 5, having a 5-6% error from true 95% UCL(Upper Confidence Limit) at a sample size of 30 and decreasing fairly quickly.

The other rules seem even more conservative. I’ll have to investigate these more closely.

So here’s my revised table:

|*Occurrences in study*|*Rule*|*95% Confidence Interval for estimated occurrence*|*How calculated*|*Old Rule*|*Asy. error*|
|0|3|0 – 3/n|mathematically|3|0.16%|
|1|4.75|0 – 4.75/n|mathematically (closer to 4.749)|5|5.42%|
|2|6.3|0 – 6.3/n|empirically — 5% error for _n_=40|7|11.21%|
|3|7.754|0 – 7.754/n|empirically — may not be conservative, 5% error for _n_=45 or so|9|16.10%|
|4|9.154|0 – 9.154/n|empirically — may not be conservative, 5% error for _n_=50 or so|10|9.28%|

So, this list may not be as memorable as 3,5,7,9,10. But do note that the rule of 9 can better be called the rule of 8. If you don’t care that much about conservativism (if you’re just trying to do these in your head, for example), then 3, 5, 6, 8, 9 will probably work pretty well.