Statistical critique: where do we draw the line? (An application to drug safety analysis)

I just read an interesting entry (and thread) on Andrew Gelman’s statistical blog that goes along the lines of some questions I have been pondering lately. Specifically, these two paragraphs hit me (this is form an email to Gelman):

The whole spirit of your blog would have led, in my view, to a rejection of the early papers arguing that smoking causes cancer (because, your eloquent blog might have written around 1953 or whenever it was exactly, smoking is endogenous). That worries me. It would have led to many extra people dying.

I can tell that you are a highly experienced researcher and intellectually brilliant chap but the slightly negative tone of your blog has a danger — if I may have the temerity to say so. Your younger readers are constantly getting the subtle message: A POTENTIAL METHODOLOGICAL FLAW IN A PAPER MEANS ITS CONCLUSIONS ARE WRONG. Such a sentence is, as I am sure you would say, quite wrong. And one could then talk about type one and two errors, and I am sure you do in class.

So, let’s consider the drug safety problem in light of this. I’ve noted before that strictly following the rules of statistics in analysis of drug safety will in too many cases lead to an understatement of the risks of taking a drug. However, we have to say something with regard to the safety of a drug, and, given the readiness of the lawyers to file lawsuits for the adverse events of a drug, it had better be correct. We do have to do the best we can.

On the other hand, let’s look at the studies looking at the autism-thimerosal connection. Both those in the camp of suggesting such a connection and denying the connection all have their methodological flaws (which makes them all the more confusing), but some of them come to the right conclusion.

Ultimately, every study has its flaws. There is some factor not considered, something not controlled for, some confounding issue. Exactly when this invalidates the study, however, is not an easy issue.

Technorati Tags:

%d bloggers like this: