Statistical commentary on the Geiers’ latest paper, Part IV


In “Part I”:http://www.randomjohn.info/wordpress/2006/03/03/statistical-commentary-on-the-geiers-latest-paper-part-i/, “Part II”:http://www.randomjohn.info/wordpress/2006/03/03/statistical-commentary-on-the-geiers-latest-paper-part-ii/, and “Part III”:http://www.randomjohn.info/wordpress/2006/03/03/statistical-commentary-on-the-geiers-latest-paper-part-iii/ of this series I discussed the statistical methodologies in the recent paper by Mark and David Geier, who extracted data from the VAERS(Vaccine Adverse Event Reporting System) and the CDDS(California Department of Developmental Services) and tried to show that efforts to remove the compound thimerosal from vaccines have resulted in a decrease in new autism cases (and other neurodevelopmental disorders). In Part I, I concluded that their statistical methodology was invalid and unable to support their conclusions. In Part II, I suggested that they could make their point more soundly by employing better, time-series-related methodologies. In Part III, I briefly examined their CDDS(California Department of Developmental Services) data, and concluded that the methodology was invalid, and correct methodology did not back up their claims. In this final part, I examine data quality issues and wrap this series up by examining a few other criticisms of their work.

h3. Data quality

“Orac”:http://scienceblogs.com/insolence and “Tara”:http://scienceblogs.com/aetiology have both questioned the use of the VAERS(Vaccine Adverse Event Reporting System) data to draw any meaningful conclusions. For example, Orac claims:

bq. [The Geiers’ paper] is yet another example of the Geiers mining the Vaccine Adverse Events Reporting System (VAERS(Vaccine Adverse Events Reporting System)) database for purposes for which it was never designed…

Orac even claims the database is “corrupted”:http://oracknows.blogspot.com/2006/02/how-vaccine-litigation-distorts.html.

Tara comments:

bq. I’ve not read the study yet, but there’s already some commentary here pointing out some flaws. I’ll add another I can see just from the abstract: they again used the VAERS(Vaccine Adverse Events Reporting System) database.

And, finally, here’s what we have on the VAERS(Vaccine Adverse Events Reporting System) database page:

bq. When evaluating data from VAERS(Vaccine Adverse Events Reporting System), it is important to note that for any reported event, no cause and effect relationship has been established. VAERS(Vaccine Adverse Events Reporting System) is interested in all potential associations between vaccines and adverse events. Therefore, VAERS(Vaccine Adverse Events Reporting System) collects data on any adverse event following vaccination, be it coincidental or truly caused by a vaccine. The report of an adverse event to VAERS(Vaccine Adverse Events Reporting System) is not documentation that a vaccine caused the event.

So, here are my comments.

On occasion, I work with the AERS(Adverse Event Reporting System), which is designed to monitor pharmaceuticals in the same way the VAERS(Vaccine Adverse Events Reporting System) is designed to monitor vaccines. This is an area that is fraught with peril. First, as mentioned on the VAERS(Vaccine Adverse Events Reporting System) page, the database says nothing about causality. Furthermore, it says nothing about the prevalence of autism, or anything else for that matter. For all those fancy statistics I did (and I basically took this as an excuse to exercise some of those statistical routines I haven’t touched in a while), the only (tentative) conclusions I can make are that the number of new cases reported seemed to rise around 1999 and may be starting to fall somewhat. Of course, the relationship between this and the removal of thimerosal is unclear, and, while the thimerosal hypothesis does predict a downswing in new cases, just because the downswing happens doesn’t necessarily mean that thimerosal is behind the issue. It simply means that the analysis doesn’t blow the thimerosal hypothesis out of the water.

The CDDS database seems to be a different matter. While restricted to California, which may have its own reason other than thimerosal usage to differ from other states in neurodevelopmental disorder prevalence, the database does seem to be a higher quality. The information is reported by California’s departments and nonprofit agencies that are regulated to deal with developmental issues in the state.

Now we have a situation where correct methods on questionable data lead to a positive conclusion, and correct methods on less questionable data lead to a negative conclusion. Of course, there may simply not be enough time to get a strong signal from the databases. However, at this point, I really can’t say that the Geiers’ hypothesis is supported in their study.

h3. Data extraction and manipulation

I didn’t touch data extraction and manipulation methods. The Geiers’ removal of duplicates (same ID) seems to be a reasonable thing to do. However, they also did the following:

bq. An ecological method was employed to evaluate NDs reported following immunizations, including autism (Costart Term = Autism) and speech disorders (Costart Term = Speech Dis), among children 5 years old). Descriptions of these adverse events by those reporting them were coded by VAERS technical staff into defined symptom fields.

Unfortunately, they do not describe their “ecological method” nor reference it. I’d rather that they did. This damages the credibility of the data.

I’m not going to be as harsh as Orac or Tara (mostly because they seem to have some vitriol reserved just for the Geiers, or at least Orac does), but I will say that any tenuous conclusions that one can come to with a sound statistical analysis are further weakened by the opacity of their treatment of the VAERS(Vaccine Adverse Events Reporting System) data. However, I will not go so far as to say that VAERS(Vaccine Adverse Events Reporting System) is not “minable.” I will say that any conclusions drawn from it are preliminary. Sayeth the Geiers:

bq. From data presented here and other emerging data, it appears clear that additional research should be undertaken concerning the effects of mercury exposure, particularly from TCV(thimerosal-containing vaccine)s.

Additional research, indeed.

h3. Other criticisms

Other criticisms seem to include the following:

* The Geiers are just out to make money from the vaccine compensation program (or some other Geiers are slime remark)
* They published their paper in a dubious journal
* They do shoddy science
* Why can’t they hire a biostatistician
* They support an untenable theory

I don’t like criticisms that appeal to motivation, and I try to avoid them myself. Sometimes they are appropriate especially when motivations are concrete and provable (such as someone’s statements).

Likewise, I don’t care that work is published in a “dubious” journal. Granted, I treat works published in “prestigious” and “lower-tier” journals differently, but not so far as to say one is right and another wrong. Up front, this says little about the actual truth or falsehood of the conclusions.

I did pretty much defang their statistical analysis in this one instance, but that doesn’t mean that they did shoddy science. I believe they are doing important work, even if they are financially compensated for it. (I’m being compensated for the work I do.) I am also going to treat their claims with an open mind — perhaps they are true or false. I’m going to try to avoid claims of shoddy science for anyone on either end of the maverick/textbook science continuum until I find some very compelling evidence otherwise (and good reason to make the accusation).

Finally, I don’t believe they are supporting an untenable theory. I think the thimerosal/mercury/neurodevelopmental disorder issue is more complex than whether thimerosal is a/the sole cause of autism, and certainly more complex than what people on both sides of the thimerosal fence often claim. This issue certainly isn’t closed, and given what I know about the state of technology to detect populations that may be sensitive to mercury, I doubt this issue will be closed for some time.

Advertisements

4 Responses

  1. Good work. Have you been keeping up with the latest on the likely retraction of this paper, and also Inververbal’s critique of the same?

  2. I haven’t looked at Interverbal’s site at some time, even though I know (s?)he did some similar analysis of the CDDS data.

    And I’ve heard nothing of a retraction. Since this isn’t directly related to what I do (and I don’t have to live with autism day in and day out), I don’t keep an ear to this ground as often as I probably should. Do you have any details? (You should be able to email me, randomjohn@mac.com.)

  3. […] If the link exists, does it vindicate Dr. Wakefield? If the charges brought against him are true, then I say that Dr. Wakefield’s conduct is still unprofessional in spite of the truth. The charges state that Dr. Wakefield stood to profit directly from the results, and that he improperly conducted the research. Tobacco science is tobacco science, whether the results are correct or not. If Dr. Wakefield wanted to further his case and retained credibility, he should have used proper research techniques, obtained proper consent from ethics boards, and obtained proper, objective funding for the work. (I’ll admit the possibility that the charges against Dr. Wakefield are not true, but time will bear that out.) I had a similar complaint (albeit not one serious enough to call misconduct) against David and Dr. Mark Geier: their methods simply did not match their research hypothesis. While I’m open-minded enough about the hypotheses (MMR and thimerosal) to ask tough questions to my doctor on vaccination day and consider alternative vaccination schedules, when you are doing science, you should use the methods of science properly. Only then can you get the credibility of having used the scientific method to prove your point. […]

  4. […] I think that we should question the establishment about our vaccine strategy. I understand the need for “herd immunity,” but I wonder if we are asking too much of our little babies’ immune systems in the process. We should always try to find a better way, one with more efficacy, less risk, and/or lower cost. Two high-profile researchers that are questioning the establishment are now under question themselves, and, unfortunately, this time, I can’t say it’s a case of The Empire Strikes Back. I believe that those who use science to back up their positions should use the methods of science properly and ethically. And this is why Kathleen’s entry, assuming the details it contains are true, is very disturbing, especially considering the methods used to obtain the data for the manuscript. There is definitely smoke here. Is there a fire? […]

Comments are closed.

%d bloggers like this: