Return on investment for clinical trials

A paper in ??The Lancet?? (and reported here) noted that the ROI (return on investment) for 28 clinical trials done by the NINDS(National Institutes of Neurological Disease and Stroke) resulted in a calculated societal benefit of $15 billion. That’s billion with a B. And if you don’t like that measure of societal benefit, try an estimated 470,000 years of life. Compare this to a $335 million (with an M) budget.

Now, this is a news report on an medical article. The issues of selection bias (perhaps NINDS(National Institutes of Neurological Disease and Stroke) have the highest ROI(return on investment) on the planet), methods of calculating estimated years of life saved, and so forth have not really been covered in any detail. I’ll also admit the possibility, as well, that ROI(return on investment) is a lot higher in NINDS(National Institutes of Neurological Disease and Stroke) research than in private research, where I work.

However, with the pharmaceutical (rightly, wrongly, or different) taking a beating in recent years, it’s very important to remember why we do what we do. Yes, we do need more transparency in how we conduct private medical research. Yes, we need to keep ethics at the forefront. Yes, we need to market and prescribe drugs appropriately after they come to market. There have been several high-profile instances in the recent past where these have come into serious question. But the thousands or perhaps millions of people involved in clinical research are still here, working hard to save lives.

Technorati Tags: research, statistics, drugs

Lying with statistics: when statistics can’t tell the truth, or why I’m interested in the statistics of drug safety (and an application to the thimerosal-autism controversy)

Drug safety is hard to study. There are so many things that can go wrong with the human body. To statistically analyze every single possible thing that can go wrong is impossible. There are thousands of possible adverse events, a whole lot of laboratory measurements that have to be taken (so we can address, among other things, whether the drug is hurting the liver, heart, and kidneys), physical exam measurements, vitals (blood pressure, temperature, respiration).

Even if you don’t find an adverse event, you still are analyzing the thousands of possible ones. They all simply have a frequency of 0, which means that the upper 95% confidence limit on a single event is 3/n (3 divided by the sample size of the treatment group of the study). To adjust this number for multiple comparisons (essentially, we have to divide 5% by the number of comparisons, though fancier methods are available), we’d have to find very close to a 100% upper confidence limit for each adverse event — almost 100% unless you include millions of people in the study!

Clearly, closely adhering to the rules of statistics isn’t going to get anyone very far in drug safety analysis until we develop new methodology.

Fortunately, new methodologies are being developed to address these issues, such as Bayesian and graphical methods. However, they are still in the cooker and probably will not be in widespread use for some time. For now, we are stuck with thousands of lines of AE counts, laboratory measure averages, vitals averages, and, if we’re lucky, a few useful graphs for labs and vitals. (Admittedly, I think simple box plots, scatterplots, and line graphs should be used more.)

When I taught first-year statistics many years ago, I tried to impress on my students that because a hypothesis test fails to show a significant effect doesn’t mean no effect is there. However, it’s usually the more conservative option to say the effect isn’t there, if a decision has to be made on the basis of the test.

In drug safety, this argument doesn’t work. To make a claim that a drug is safe, we have to say that it does not cause more adverse events and does not cause unsafe laboratory or vitals findings. The more conservative statement is to say that a drug does cause an adverse event. However, this will essentially lead to the statement that the drug might be too unsafe to use. (How would you like to say that, while no incidence of torsade de pointes was observed, the clinical development program wasn’t robust enough to say that the drug doesn’t cause it?)

So the reality of the current situation, and the state of the art of drug safety analysis, is that we statisticians generate thousands of lines of results and the pass it off to one or more medical writers who try to make sense of it all. (And they usually do a good job, although the FDA has been known to require warnings on the label for events that have occurred only in one animal in only one preclinical study, even when it doesn’t occur at all in the clinical studies.) We statisticians can do better, and we are starting to do better, but right now the 1960s is where we are with safety analysis.

Incidentally, this is why I don’t hold statements like the following in high regard:

I want to be as clear about this as I can. There is no controversy
surrounding Thimerosal. There is scientific evidence and there is
hysteria. The scientific evidence suggests that there is no link
between thimerosal in vaccines and autism or any bad outcome whatsover!

By now you should know what what my response is: if Dr. Flea is going to make such a strong assertion, I expect a tractor-trailor full of CDs full of compressed PDFs with studies disproving any link between thimerosal in vaccines and “autism or any bad outcome whatsoever.” If you want to know the reason the thimerosal-autism story will not die, here it is. Because it’s darn near impossible to collect enough scientific evidence to disprove the link, so anecdotal evidence is going to keep the questions rolling. Which I say is a good thing for the most part, despite the recent (possibly valid, possibly invalid) allegations against two groups of researchers investigating the harmful effects of vaccines.

At any rate, this is why my recent interest has turned toward the analysis of drug safety. Because it’s a hard problem.

Acupuncture vs. fibromyalgia: 2 out of 3 isn’t bad

A recent study found acupuncture to be an effective therapy for treating fibromyalgia pain, fatigue, and anxiety. Of two other studies, one reported positive conclusions and one reported negative. The study claims to be a model of how acupuncture clinical trials can be run.

Technorati Tags: acupuncture, alternative medicine

Music: the undrug

Music has been the subject of a recent volume in the Annals of the NY Academy of Sciences.

I’ve recently covered music as both a therapy for asthma and diabetes and as synergist of sorts with MDMA (ecstasy). Frank Herbert knew about the power effects of music when he wrote in ??Dune?? about the drug and music combination (mostly for mind control).

Think of how you can use these facts to set your own environment.

While anti-vaccine researchers are being charged with ethical lapses, the mercury issue marches on

Dr. Mercola links to an public service announcement about the presence of mercury in vaccines. This PSA(Public Service Announcement) contains one tidbit that I haven’t heard yet: that the EPA suggests that the amount of mercury still present in vaccines is safe only if you weigh over 500 pounds. Is there a source for this information?

Update: So, I’ve dug a little deeper. Here’s what I’ve found:

• The EPA’s webpage on human exposure of mercury is here.
• I found the following statement in mercury’s tox profile:

EPA and FDA have set a limit of 2 parts inorganic mercury per billion (ppb) parts of water in drinking water. EPA is in the process of revising the Water Quality Criteria for mercury. EPA currently recommends that the level of inorganic mercury in rivers, lakes, and streams be no more than 144 parts mercury per trillion (ppt) parts of water to protect human health (1 ppt is a thousand times less than 1 part per billion, or ppb). EPA has determined that a daily exposure (for an adult of average weight) to inorganic mercury in drinking water at a level up to 2 ppb is not likely to cause any significant adverse health effects. FDA has set a maximum permissible level of 1 part of methylmercury in a million parts (ppm) of seafood products sold through interstate commerce (1 ppm is a thousand times more than 1 ppb). FDA may seize shipments of fish and shellfish containing more than 1 ppm of methylmercury, and may seize treated seed grain containing more than 1 ppm of mercury.

• John Hopkin’s University (Bloomberg School of Public Health) maintains a vaccine safety site. They included a table of thimerosal concentrations.
• The FDA maintains its own thimerosal page. Of note, according to a 1999 review, “…, depending on the vaccine formulations used and the weight of
  the infant, some infants could have been exposed to cumulative levels
  of mercury during the first six months of life that exceeded EPA
  recommended guidelines for safe intake of methylmercury.” (The Hg-containing metabolite of thimerosal is ethylmercury.)

If you do the math on the 0.1µg/kg/day reference dose set forth by the EPA and compare it to the tables on the FDA page and Vaccinesafety.edu pages, at least for the pediatric vaccines, you don’t get 500 pounds. The worst case scenario that I found was the Fluvirin® that was before 9/28/2001 and Fluzone® flu vaccine that was before 12/23/2004, both of which contained 25 µg of thimerosal in a 0.5 mL dose before newer versions were approved. This does work out to a little over 550 pounds (25 µg / 0.1 µg/kg =250kg = 551 lbs), and this may very well be the source of the 500 pound message; however, the vaccines have been replaced with one with one less than 1 µg/0.5 mL dose or even thimerosal-free versions.

Mark and David Geiers’ IRB: If there is a story here, I’m disappointed

Activists asserting a connection between vaccines (or components of vaccines) and autism have had a hard time of late. First, it was Dr. Wakefield, proponent of the theory of the connection between the MMR(Mumps, Measles, Rubella) vaccine and both autism and irritable bowels, was formally charged with professional misconduct. Now, Kathleen Siedel of Neurodiversity has dug up some disturbing information on Dr. Mark and David Geier. This has to do with the creation of an IRB(Institutional Review Board) that oversaw the protocol that resulted in the recent manuscript A Clinical and Laboratory Evaluation of Methionine Cycle-Transsulfuration and Androgen Pathway Markers in Children with Autistic Disorders. This paper dovetails with their Lupronâ„¢ (i.e. chemical castration) strategy for treating children with autism.

Repeat readers of this blog know that I am agnostic on the thimerosal-autism connection hypothesis. I even have my doubts about the safety of the MMR(Mumps, Measles, Rubella) vaccine. The complexity of the mind is such that we simply don’t understand how these things work, and even running tests for mercury in the blood isn’t easy. And the vigor with which people on both sides of this controversy argue seems to leave little room for real understanding.

Lying with (non)statistics: Phasers are set to “kill habit”

Public citizen has “filed a petition”:http://www.wral.com/health/9412806/detail.html to end the marketing of a “low-power laser acupuncture” treatment for smoking cessation.

What struck me was not the treatment itself, but the way in which the so-called clinical trials they are running to “prove” their treatment’s efficacy. You can read about it at the link, but to me the following details are very fishy:

• the company charged smokers to participate in trials (it’s usually the other way around — companies usually pay participants) — in fact, companies charged more to participate in their trials than GlaxoSmithKline does for a month’s supply of Zyban
• the company did not collect any direct efficacy data, such as the time to smoking cessation (or whether a person stopped smoking at all) (in which case the trial is unethical)
• the company is conducting trials, but is claiming their efficacy measure is “client referrals”
• “When we do follow-up phone calls, people don’t call us back,” according to one product’s owner. You know what? If you are going to conduct a clinical trial, you have to control for things like this. Of course, there’s always going to be people who escape tracking, but in the clinical research industry we’ve become very good at designing studies so that people can be followed up. It’s a heavy cost, but well worth it and even necessary to conduct proper clinical research.

Since smoking cessation is a huge market, and effective non-pharmaceuticals is probably very desirable, I can only imagine that a patented medical device with lots of properly-run clinical trials to back it up can make a lot of money.

Another company believes it has enough data to submit to the FDA, but was cited a year and a half ago for failing to properly monitor its trials. And take it from this biostatistician, if a trial is not properly monitored (i.e. someone verifies that the data has been recorded correctly and that the correct data was recorded), it’s hard or next to impossible to make any accurate conclusions. Any results from that trial is weak. For their sakes, I hope that they have several other well-run studies that support their application.

Who knows, maybe the theory of acupuncture works with lasers, too. Maybe endorphins are released when you nail someone with a laser. It shouldn’t be too hard to find out these days. If you want to use the methods of science to back up a claim, use them properly, or else someone will call you out. Or even worse, someone will write you up for a skeptic’s circle post.

Every high school needs one of these

I probably needed to be sent to this room several times during my grade school years.

Spontaneous mountain hiking

Ok, so after yoga yesterday and some time relaxing at home, the family and I decided to try out a “mountain” state park close to our homeI put mountain in quotes because the elevation at peak is 867 ft above ground level, at least according to the literature they have. I understand that in order to qualify as a mountain it has to be 1000 ft. However, it was still a good time.. We were just going to go for a picnic, but then decided to follow a trail a little ways and then go back. Well, halfway around the mountain, we sat down on a bench and decided to plod further. Yes, we had two children with us. Yes, this was insane.

At any rate, three-fourths of the way around we took a wrong turn and, after going under a high-tension power line tower, we came to a rather pretty area with exposed rock. We were clearly lost. So we went back up a bit and took the right way.

Total time: about an hour an a half. Just for a fifteen minute walk up the path. Clearly I miss mountain hiking. Next time, we’ll use a map.

At any rate, after the most physically grueling day in weeks, I just want to sleep all day. It’s back to work, though…

Getting back into practice

So today I went back to a yoga class for the first time in 3 weeks, and the second time in 5. I much prefer going once a week, and preferably two or three, but at least once. However, when life intervenes you do have to roll with the punches.

In the meantime, I tried to keep up with practice at home as best I could. That didn’t always work out, and now I feel a few pounds heavier. So it’s back to cutting out the sugar and bumping up the exercise — I can’t change my past behavior but I can change the present.

Practice today was an odd mixture of backing off old, familiar poses (e.g. ustrasana) which I knew would carry me over the edge of fatigue and courageously trying less familiar poses such as parsva bakasana, eka pada koundinyasana, and the twisted child’s pose. I feel very energetic right now because of the practice, and, of course, I’m aching for more.

It’s time to pick up the pieces of my home practice and build something better.