Lying with statistics? Misunderstanding of noninferiority trials leads to illogical editorials


So Dr. Mercola, following the lead of an editorial and questions raised by Sen. Grassley, has questioned the use of noninferiority trials. I quote Dr. Mercola:

According to a fascinating Wall Street Journal piece, antibiotics, unlike other drugs, adhere to a greatly diminished standard. A drugmaker need only demonstrate that its product works “about as well” as older drugs, and not necessarily better than existing ones.

Referred to as noninferiority trials, these procedures provide results that, unlike conventional placebo-driven testing, are less clear-cut. In fact, noninferiority trials have been used to judge the value of antibiotics for some 60 years.

From the tone of this quote, you’d think that we’re lowering our standards on antibiotics. And it strikes me as a bizarre argument.

Let’s think about this. On the one hand, we have a placebo-controlled trial hopefully will show a new drug superior to placebo, or we compare a new drug with an existing standard of care that has already been shown to be superior to placebo. What is not clear cut about that?

Noninferiority trials have an important place in drug development. First up, I’ll let Frank Tally of Cubist Pharmaceuticals take the mic:

For more serious infections, “it would be immoral to do a placebo-controlled trial where there is an established therapy,” says Frank Tally, chief scientific officer at Cubist Pharmaceuticals Inc.

For more minor infections, the situation is still, I think, not hard to understand. It is more than just this:

Some doctors and industry officials say that patients would be reluctant to take part in studies where they risked being given a placebo, even for relatively minor infections.

Non-inferiority trials establish efficacy that is at least as good as an older standard of care, and it’s even better if the new treatment holds some advantage over the older one. Here are some possible advantages:

  • better safety profile (i.e. fewer or less severe side effects)
  • easier to take, so better compliance, better quality of life, higher efficacy in practice
  • easier to manufacture, so less variation in the quality of the produced compound
  • higher shelf life
  • easier storage conditions
  • on and on

Here are some characteristics that might be different about a new drug that, while efficacy and safety profile may be about the same, will still have the new drug be useful on the market:

  • different mechanism of action (think of the different classes of hypertension medications) – the new drug may be more effective in populations that for which older drugs may not work, even if the overall average population effect is about the same
  • different metabolic route – this changes, among other things, the drug-drug interaction profile so the new drug may be taken with different medications from the old drug, or simply fit better with the genetic profile of some people
  • the economic argument of competition
  • and so forth.

In the field of infectious diseases, there is one more inherent argument for having more drugs on the market, even if they are equivalent in efficacy: antibiotic resistance.

Also, by focusing in on just efficacy, we are ignoring the wide array of issues the FDA considers when approving a drug. Drug development often takes a decade and anywhere from 5 to 30 trials (with most development programs aiming for just approval in the US taking about 7-10). For each of these trials, one patient can generate on the order of 20,000 data points. 50 patients will generate a million. For many drugs, the agency wants to see at least 1500 people exposed to the drug for a month before a drug is submitted for marketing approval, and often more patients than that end up in the development program. It takes a while to sift through the information, which covers safety, efficacy, drug interactions, movement of drug through the body, adverse events, laboratory values, and so forth. Often the agency will require warnings based on an adverse effect on one animal in an animal study. Also, in most cases, the agency will require two positive large “pivotal” trials before they consider approving a drug.

To be fair to this line of questioning, there are some important issues raised. For example, there are questions as to whether the FDA is allowing too large a non-inferiority margin (i.e. how much the new drug can underperform in the trial before being declared inferior). This is definitely a hard question, and it needs to be handled carefully for each noninferiority trial conducted. If you set the margin too high, you risk putting an ineffective treatment on the market. If you set the margin too narrow, and you effectively defeat the purposes of running a noninferiority trial in the first place — i.e. not needing to play a game of pharmaceutical one upsmanship in the cases where you really can’t run a placebo-controlled trial due to ethical concerns.

Selection of the older antibiotic standard is also an important issue, especially when you have the following:

FDA officials have conceded that for certain uses, including bronchitis and sinusitis, the extent to which older antibiotics beat placebos isn’t clear. An agency statistician told an advisory committee in 2002 that “historic data is often poor” in anti-infective drugs.

I think this will be an important question for public debate in the coming months.

Now, for Ketek, the issue was liver toxicity. This is a safety issue, and one that needs to be addressed. I imagine that Sanofi and the FDA will both have their share of hard questions to answer on the Ketek matter. Let’s ask the right questions, not ones like “You approved this for marketing because it was at least as good as an older drug?!”

Advertisements
%d bloggers like this: