Great, another doctor who likes to deliver “Respectful Insolence”


I’ll skip over some of the nonsense “here”:http://oracknows.blogspot.com/2005/09/little-criticism-directed-oracs-way.html and jump straight to the meat of the matter. (And forget about the _ad hominem_ attacks in the inane comments, which numbered 4 at the time of this post. It’s best that way.)

bq. (although I would disagree with his qualification of his dislike in which he said that there’s “a lot to Kevin Trudeau that’s admirable”–I consider Trudeau to be a scam artist)

At the age we were still building towers with erector sets, the boy was building businesses. I don’t know about you, but I find that admirable. Too bad he runs scams today — like most other people, I don’t find that admirable. I don’t live in a black and white world, so I can easily make comments about admirable qualities in criminals.

Oh, and by the way, I don’t waste my time on the Usenet groups. Maybe if Orac skipped the newsgroups, especially in one he obviously hates, he’d have enough time to think a little.

And, should he skip tonight’s misc.health.alternative session, he might think a little more about these points.

bq. If you accept vague and/or poorly documented anecdotes and testimonials as sufficient evidence that an “alternative” therapy “works,” you just might be an altie.

bq. If you dismiss every well-designed randomized clinical study that failed to show a benefit for an alternative medicine or therapy over placebo control as either not proving that the therapy is ineffective or as having been manipulated by nefarious forces (conventional medicine, the pharmaceutical companies, the government, etc.) to produce a negative result, you may well be an altie.

Straw man, anyone?

Now, let’s take this even further. Say there were two candidate compounds: the pharmaceutical Fixitol and its competitor Herb Remedy. An adequate well-controlled randomized trial shows superiority of Fixitol over Herb Remedy. In addition, Ms. Jones up the road swears up and down that Herb Remedy cured her illness, and is willing to be quoted in the newspaper for saying it.

Who’s right?

Let’s examine Orac’s answer:

bq. Uh, no, John, it’s not “bias” to understand that testimonials are inherently less reliable than clinical trials in identifying which treatments work and which do not and then putting them into their correct place in the hierarchy of medical evidence. It’s just good science. And, no, John, I did not say that anecdotes are “bad.” I merely pointed out that they are far less likely to be widely applicable than the results of clinical trials.

Let’s see how less widely applicable anecdotes are: Nozitol was an approved compound for 3 years. It is blamed for 40 deaths, and on the basis of that, was removed from the market.

Not so simple, eh? Thousands of people tested in clinical trials, and 40 suspected deaths results in a wider application than all those other people. Oh yeah, let’s reveal the true name of Nozitol: Baycol.

bq. Finally, contrary to John’s characterization of what I said, I have not entirely dismissed anecdotal evidence. Actually I have said repeatedly, that such evidence, if better documented than just a testimonial, can be useful to guide further research.

At least we have some agreement.

bq. Testimonials are often highly subjective, and, of course, practitioners can and do pick which testimonials they present.

Ok, to be fair to Orac here, I am picking and choosing which statements to respond to. And, we might point out that he’s doing the same to me. I’m picking which points of his to which I’m crafting a response, and he’s, well, picking which points of mine to which he crafts a response. Hmmmm… this is starting to sound a little subjective.

I challenge Orac to prove for once and for all that, if he were able to wave a magic wand and remove testimonials from the pharmaceutical process, that it would be completely objective. In fact, I’ll make it easier. I challenge him to prove to me that an experiment with a high-energy particle accelerator is entirely objective. Whether you choose to believe testimonial evidence is entirely up to you. Quite frankly, I’m disinclined to believe testimonial evidence, but I also have to question the worth of a product if a company cannot even pay someone to speak for it.

Ok, so maybe the standard is too hard. Because, I assure you, I’m waiting with tons of examples of where subjectivity creeps in the drug development process, and, perhaps even nastier on my part, I believe that’s ok. I think it’s perfectly fine to have subjectivity in the pharmaceutical development process, and, even more, I believe that it’s impossible, despite the advances of my field, to expunge it.

Now, taking back the magic wand, will you go down to the marketing departments of Pfizer, Merck, Bristol-Meyers-Squibb, and GlaxoSmithKline and tell them to stop using testimonials? (And even their regulatory departments, if we’re talking drug development.) Please carry along a video camera, and put the results up on the internet.

Let’s get back to picking and choosing which points of Oracs I want to refute.

bq. John did have a point in mentioning that clinical trials may not adequately predict the response of any single individual to treatment. That does not invalidate my point, however, because anecdotes are considerably worse than clinical trials at such prediction.

Ask the ghosts of the 40[1] or so people who died as a result of the side effects of Baycol how well clinical trials predicted their response to treatment. The “anecdotes” of each one of those people is closer to the 39[1] other people than to the thousands in the clinical trials. I’d say they both are pretty bad. The whole point of the post to which Orac responded was how often this effect is minimized. I ought to make him go write the bolded part of the post on the chalkboard for 45 hours (approx the length of a 3 hour semester class) before he’s allowed to practice as an academic surgeon further.

bq. Note the last sentence. That’s me pointing out the proper weight that should be given to anecdotal evidence, which is not zero but is much less than the weight that should be given to a well-designed clinical trial. I suspect the problem here is that John seems to be equating testimonials and anecdotal data.

Please, oh Orac, saint of medical scientific research, tell us what proper weight should be given to anecdotal evidence and clinical trial evidence. If thou wouldst but give us thine formula, that we may follow it.

bq. In addition, it isn’t more anecdotes that will help us predict more accurately the response of any individual to any given therapy. It will be clinical trials that identify factors that might help us predict which individual patients will respond better to which treatment. Indeed, that’s the whole point of genomic medicine and molecularly targeted therapeutics like Herceptin.

Ah, a vision! Thank you, oh Orac, for this wisdom!

Let me go one up on you: it will be anecdotal evidence, perhaps taken during a clinical trial, that will identify some subpopulations that benefit from a particular therapy. It will be a clinical trial that provides the evidence that the subpopulation really does benefit. And then when the drug is marketed for that subpopulation, some people will respond in completely unexpected ways and some people will respond within the 95% confidence interval. And the process will repeat. In the meantime, people will still benefit, and people will still be harmed by, conventional and alternative therapies.

Now that we got that out of the way, let’s get back to alternative vs. conventional medicine (if there is ever such a competition). I have a list of questions:

  1. How many people has conventional medicine cured?
  2. How many people has homeopathy cured?
  3. How many people has Reiki cured?
  4. How many people have African shamans cured?
  5. How many people has osteopathy cured?
  6. How many people has witchcraft cured?
  7. How many people has stoning people cured?

I’ll give you a hint: the answers are all the same, and these are all trick questions. And in fact you can put in any healing system. It doesn’t matter. Healing systems don’t “cure” anybody. At best, they help the body heal itself, and that’s the way it should be and the only way it can be. In some miraculous cases, there’s help from parts of another person’s body (although that’s dangerous and forces us to suppress the body’s own healing capabilities).

Well, do have a lot more to say about this:

  • Assumptions of statistics, which powers the evidence part of a clinical trial
  • More about the distinctions between testimonials and anecdotal evidence
  • More on the “my health care system will kick your health care system’s ass” syndrome
  • More on not living in a black and white world
  • More on why Robert Carroll’s system of skepticism has a lot of bullshit (which is the whole point of the skeptic’s skeptic’s dictionary category)

And I might have more time in the future to say it, and hopefully in a much more coherent way than I made my points here.

fn1. Lest you get picky and jump on the number 40, I fully recognize that there are undocumented cases of Baycol deaths, documented cases where there may have been predispositions to rhabdomylosis, and so on. It’s simply the best estimate I’ve seen. And it’s small compared to the thousands who took it during Baycol’s development, and behind this number is are people who died miserably and families who grieve their deaths.

Advertisements

5 Responses

  1. Orac is among the self-appointed defenders of science who love to quote the old saw about the plural of anecdote not being evidence. I’m all for science, but I would just as soon not have Orac define what science is. He generally refuses to admit the possible validity of any hypothesis that has not been definitively proven. My less-than-perfect recollection of the scientific method is that it begins with “observation,” into which category anecdotes fall. In my estimation, even an unproven hypothesis — if it is supported by statistically significant anecdotes — remains potentially viable until disproven.

  2. It gets even worse. It’s not an “anecdote” unless some elite guardian of science has approved it. Take a look at his “diatribe”:http://oracknows.blogspot.com/2005/09/little-criticism-directed-oracs-way.html, if you can stomach it.

  3. John, I think I love you.
    Are you married? LOL!

    Keep up the great blog writing.

  4. *blush*

    Yes, and I’m about to have my second child.

  5. […] Pat Sullivan, Jr. discusses in some recent posts the roles of different types of medical evidence. First, let me get an issue out of the way concerning the definition of “anecdote,” since it seems to get in the way. From Sullivan’s entry: […]

Comments are closed.

%d bloggers like this: