Ranting and raving again about healthcare and the magical clinical trial

Posts like this simply make my blood boil. Perhaps I should keep my “buttons on the inside” (i.e. not let other people push my buttons), but I get frustrated when doctors get their priorities all out of whack and misinterpret the results of experiments. Let’s try this one more time:

Statistics measure population tendencies, and are bad at predicting individual results. I ought to make every doctor on the face of this planet write this on a chalkboard until they say it in their dreams. Maybe it should be a semester course in medical school and biostatistics. A three-hour course required for graduation. Three hours a week of writing this on the board, supervised, and a recorder stuck under their mouths at night for verification. To pass the class, they have to be saying this in their sleep.

Orac seems to take the most extreme example of the alternative-medicine-is-good-and-conventional-medicine-is-bad nutcase (and, the person who fits the description he gives on the page is a nutcase, I’ll give him that), and equate people giving testimonials.

Let’s look at this argument then: clinical trials good, testimonials bad. Do you see the bias here? No? Well, let’s give a different example:

Testimonials good, clinical trials bad. Do you see the bias here? Ok, good. Now, let’s go back: Clinical trials good, testimonials bad? Oh, still don’t see it? Let’s phrase it a different way: Evidence of population tendency good, individual people’s results bad. See what I’m getting at? No? I’ll show you the link again: here. Read it again. And again. And I’ll join you when you’ve got it down.

So testimonials and anecdotal evidence? I’m glad the FDA hears them. We need to hear them. After all, they come from the people that we in the healthcare/pharma/research industries are trying to help. The sparks for ideas that lead into scientific revolutions come from odd, anecdotal observations that are outside of what statistics predicts. Individuals give important information that statistics will miss, and cutting out anecdotal or testimonial information and relying solely on clinical trials for our research is like cutting off our legs because the car gets us there faster.


17 Responses

  1. John, At the risk of “pushing your buttons” I was always under the impression that anecdotal evidence wasn’t in fact evidence. Let me give you an example I like to use. George Burns smoked 4 or 5 cigars every day and lived to be 100. Does that mean smoking cigars is harmless? And asking doctors to accept anecdotal evidence is also asking them to climb way out on a liability limb.

  2. Ok, let’s take your example. George Burns smoked 4 or 5 cigars a day and lived to be a hundred. Does this mean that, even with the statistical models showing a strong correlation between smoking and reduced lifespan, here is an example, perhaps extraordinary, of an individual that lies way outside the bounds of the model. Rather than accepting the anecdotal evidence and saying that all people can smoke 4 or 5 cigars a day and live to be a hundred, perhaps there’s something in Burns’s case that challenges some of the assumptions behind the models. Perhaps there’s a gene he had, or maybe he didn’t have any of the habits that commonly accompany smoking, or whatever.

    Does this change the doctor’s advice to quit smoking? Absolutely not. But it should prompt a further investigation into this relationship between tobacco smoking and lifespan. It’s not as simple as the statistical models, however strong they are, suggest.

  3. I agree that in the George Burns model there would be cause for further investigation, but that is research. Doctors by and large aren’t researchers. They are clinicians. That means they are going to want proof it works. Anecdotal evidence just doesn’t fit the bill.

  4. The best doctors I’ve seen had a bit of the researcher in them. They recognized that yeah, our bodies are very similar and so there can be a medical body of knowledge based on research and statistics, but that’s not the whole story. Each person is an individual, and so the medical body of knowledge, even when it suggests investigation, isn’t going to work for everybody.

    By contrast, the worst doctors I’ve ever encountered, or, should I say, my worst experiences with the medical profession involved doctors who took the medical body of knowledge to be the absolute truth, that everybody’s bodies were more or less described adequately enough that recommended courses of action were the only ones that needed to be considered. They weren’t researchers at all, nor were they curious.

    For example, my daughter had problems with recurring ear infections. The first doctor’s response was to give antibiotics for each recurrence. After a couple of times of this nonsense, we decided to take her to a homeopath. The homeopath made a very thorough investigation into my child’s condition, including behavior, responses to food, the whole 9 yards. She then prescribed a remedy. After a couple of weeks of the remedy, we reassessed, and and adjusted. The symptoms disappeared, and then a month later reappeared. The homeopath mentioned this was a good sign, and recommended that we take her off the remedy. So far we’ve enjoyed the longest period of time symptom-free.

    Will everyone share this experience? Absolutely not. Does this (anecdotal) example “prove” homeopathy? Hell no, and it isn’t intended to. For some, perhaps many, homeopathy isn’t the way to go. However, blind faith in conventional medicine, homeopathy, this or that way of doing things is all counterproductive. It closes our minds to the mysteries of our bodies and opening up to the truth behind our bodies’ reactions to our environment.

    Any doctor that requires “proof” that something will work to the exclusion of curiosity about what might be at work is one that I’m going to avoid, if possible.

    Unless our stupid approach to medicine forces doctors to all become the same through malpractice insurance.

  5. And here you hit upon the problem. Let’s say I come up with a treatment that does
    work for some people, but it doesn’t have any proof of efficacy. The first person who is treated that doesn’t get better is going to sue the doctor’s butt off. And probably win.

  6. It’s a wonder all homeopaths, chiropractors, and acupuncturists haven’t been sued out of business.

  7. Having read Orac’s blog for a long time, I don’t see his position to be that testimonials are “bad”. Testimonials often make a good starting point for further research, but they aren’t reliable sources of information if you’re contemplating the widespread use of a medical (or alternative) treatment for a specific condition.

    Testimonials can be easily messed up by false-cause errors that good clinical trials are designed to eliminate. Furthermore, a reliance *exclusively* upon testimonials is a strong sign of charlatanism.

  8. John,

    You took your daughter to a homeopath, they “prescribed” a remedy, after 1 month she had another ear infection, so you stopped the
    remedy any she hasn’t had an ear infection in a couple of months – and this is a success? This is exactly why a testimonial or anecdotal -based medicine practice is less effective than evidence-based medicine. You conviently ignored the negative effect of the remedy (recurrance of ear infection) and when the remedy was discontinued, her lack of recurrance (again) was blamed on a remedy no longed used.

    If this is what a successful test looks like to you, then I hope that you are not the person designing or analyzing the results for your pharmacutical company.


  9. Cheeto, it appears to me that you don’t understand the way homeopathy works. That’s fine as far as that goes, but I recommend you do a little more research before belittling me or my opinions and experience. Maybe you’ll run across the theory behind why that sequence of events was considered a success. Here’s a hint: it’s not based on the same assumptions as conventional medicine, and I have a hunch that unless you set those aside for a moment you won’t understand. (And if you do set those assumptions aside maybe you will.)

    And guess what, I do design and analyze clinical trials, and a lot of people trust my work, and I also happen to believe that my little corner of the world isn’t the absolute truth to the exclusion of anything else that may be out there. I’ve also seen a lot of people with terminal degrees screw up expensive trials because of lack of understanding about what they can or cannot do.

    Runolfr, I agree with you, and will add one thing: Furthermore, a reliance exclusively upon clinical trials is a strong sign of inflexibility. Clinical trials are designed to eliminate false-cause errors, and they’re pretty good at it, but they aren’t perfect, either. (And, given the practice and constraints you have to work with, I’d only rate them as “pretty good” and “the best we have to work with in idealized cases.”)

  10. Excellent post, my man. You said it all. Orac has been ranting on his blog about this post. I’m glad you got him all worked up. That guy needs a wake up call.

  11. Good, he needs it πŸ˜‰

    Maybe I’m hurting in this situation. I mean, doctors have the difficult job of taking information about populations and tendencies and applying them to individuals. This is not very easy. I put in a “new post”:http://www.randomjohn.info/wordpress/2005/09/29/great-another-doctor-who-likes-to-deliver-respectful-insolence the story of Baycol, where the experience of about 40 people was radically different than that of thousands of other people. (They died from rhabdomylolysis and related conditions.)

    Still, as a father and someone who knows a thing or two about the methodology behind the science, I have a responsibility to use my knowledge the best way I can to help my family. The first is by realizing what exactly FDA approval of a treatment means. It means that it has been through a certain process designed to show safety and efficacy. The methodologies used to do this by necessity show population (or subpopulation) tendencies, and doctors may gravitate toward the ones that show the biggest tendencies (or are newer, or are more heavily promoted), the story doesn’t end there, though. There’s a lot more, and a lot of subjectivity despite the best efforts of a lot of smart people, and, as I pointed out in my new post, I’m not sure that eliminating subjectivity is the best goal here. Instead, I think that understanding the role and effect of subjectivity and using it (e.g. embracing rather than denying the placebo effect) is a far better goal.

    In short, I think individuals need to take charge of their own health and understand the role of doctors/healers and healing systems (so-called Western medicine included). The pharmaceutical and healthcare industries are populated by people who have the best of intentions (and a few with the worst), and who are also individuals with an individual point of view they’ll never be able to escape even with a million science textbooks and yapping in blogs about science and evidence and pointing to Skeptic’s dictionary to “prove” their points.

    Ok, I’ve really said enough here. There’s a lot of ground to cover here, and I’ve got clients and a baby who’ll be here any day. “Commenting in my own blog” is a terrible excuse. πŸ˜‰

  12. LOL! John, your frustration sounds so-o-o-o-o-o familiar!! But thanks for trying to inject some grey into Orac’s black and white world.

    As a statistician working for Pharma who also believes the “natural cures” (couldn’t help being facetious) can actually work, I find your arguments to stem from an interesting place. I would actually be very interested in your take on this post Autism/Mercury Connection – How much evidence is enough? and the article “On Evidence” that it references.

    PS – I’ll warn you in advance that your baby is likely to be born before you can wade through all the Autism-Mercury/Orac arguments our two blogs. But good luck with it!

    – Patrick Sullivan Jr.

  13. Ok, html tags don’t seem to work for your comments. The link I added about “Autism/Mercury Connection – How much evidence is enough?” is supposed to go to this post:

  14. […] Thanks to commenter Pat Sullivan, Jr. in the rant on clinical trials, I’ve finally found out who Orac is, and it seems I’ve stumbled into a bit of a hornet’s nest. Quite frankly, though I’ve viewed the Quackbusters for a long time as an example of the worst that science can produce (given that, in addition to science and quality of life improvements, science generates territorialism, credentialism, and a whole host of other things that are neither positive nor “scientific”), I never thought I’d be tangling with one of them. But you know what, if this continues I’m not going to back down, and here’s why. […]

  15. Pat, if you type out the URL without HTML tags, WordPress will automatically convert it to a link for you. I’ve also turned on Textile 2.0, because I use that syntax for my posts.

    I’ve created new posts to address what you’ve brought up, but be warned that I’ve only read very little of your thimerosal/autism posts. I’ve read “Derek Lowe’s opinion”:http://www.corante.com/pipeline/archives/2005/06/27/thimerosal_again.php on the matter (which is essentially “thimerosal doesn’t cause autism, can’t we move on now?”) and can only assume that Orac’s is similar. Maybe, in the mounds of time that I have as a father-to-be with a two-year-old also, I’ll catch up πŸ˜‰

  16. […] So, commenter Pat Sullivan, Jr. (author and CEO of Jigsaw Health? Update: Pat clarifies his role with Jigsaw Health in the comments. Thanks Pat!) has asked me to weigh in on the thimerosol debate. For the uninitiated, thimerosal is a compound added to vaccines to increase shelf life. One of its components is mercury, a heavy metal which the FDA and EPA both warn about on a regular basis in connection with pollution and fish. The debate is between the sides that believe that thimerosol increases incidence of autism and the side that insists that thimerosol is completely safe. Both sides have their salvos of studies that “confirm” one of the hypotheses, and both sides have contigents of statisticians and scientists ready to pick apart the other side’s studies in any way it can. […]

  17. […] probabilty to treat people versus using it to treat groups 28 November 2007 Another blog has an interesting post complaining about about the “magic” of clinical trials and the refusal by many people […]

Comments are closed.

%d bloggers like this: