And another thing on birth control effectiveness stats...

The more I think about these birth control effectiveness measures, the more problematic they become. I already explained in my previous post that they represent group averages, and so cannot be taken to apply to any particular individual. That might not be such a problem if these stats were used for formulating population health policy, where a statistically similar applicable population might be assumed. But in reality, by far the most common use is by individuals trying to understand their own risk, which may in fact diverge significantly from that of ‘typical use’. In that situation, you can probably trust effectiveness data in broad brush strokes (say, the overall ranking), but I wouldn’t read too much into small differences.

An even bigger potential problem though, is that unlike pretty much all other comparative therapeutic data, where a particular intervention is randomly allocated, these stats seem to be based on ex post epidemiological (primarily self-reported survey) data. That certainly seems to be the case for this study, which was apparently the source for the NYT graphics (h/t Mona Chalabi).

That means the users of each method have self-selected into its use. Whereas randomisation would ensure that 100 couples using condoms was statistically similar to 100 couples using the pill, self-selection means they could be quite different.

Just to pick a few potential confounding factors:

• Users of condoms may be systematically younger, in less-stable relationships, and more inebriated: all of which seem likely to lead to higher rates of unplanned pregnancy.

• Users of more ‘committing’ methods, like implants, IUDs and sterilization, seem likely to be systematically older, and since natural fertility falls with age, some of the higher observed effectiveness of these methods may simply be due an older user population. Not to mention that frequency of intercourse must vary systematically with age (up, then down?).

• Users who are bad at keeping habits, and know this, may systematically avoid methods such as the pill, which would artificially raise their typical effectiveness compared with a random population.

Again, this wouldn’t matter so much for public health purposes, but if you are a couple or individual trying to decide on a method of birth control, these other factors are largely out of your control and should be factored out of the decision.

Perhaps somebody who works in this field can advise: are there either fully randomised comparisons, or at the very least multivariate comparisons that attempt to account for obvious factors like age, education and seriousness of relationship?

This seems to be a problem more generally with population risk statistics: it’s often very hard to collect covariates (especially unobservables like risk attitude), so instead we’re presented with population averages and expected to make some sense of them as individuals. This seems like a massive opportunity for the ‘quantified self’ technologies.