Prospectively defined vs. exploratory analyses

Inspired by a recent “News Target”: article entitled “lying with statistics” (and a book — How to Lie With Statistics), I’ve decided to do a series on Lying with Statistics. You can find other entries in the series “here”: On with the show.

A prospectively defined analysis is one that is stated in advance, before data are collected. For purposes here, an exploratory analysis is one that is defined and specified after a view of the data. (Gee, these numbers look different between the sexes. Let’s test that hypothesis.)

Both kinds of analyses have their place. However, confounding the two leads to a grievous statistical error, and, if the researcher does this on purpose, it is a subtle form of deception.

In my industry (clinical research), our trial objectives and primary analyses have to be stated in advance, that is, spelled out in writing before we look at the data. In fact, “ICH(International Committee on Harmonisation)”:, a group that seeks to ease the regulatory burden on submitting drugs for marketing approval internationally by aligning regulatory standards, is more blunt:

bq. The extent to which the primary analysis is planned in advance of the study contributes to the degree of confidence of scientific and regulatory authorities in the final results and conclusions of the trial. (ICH E9 guideline)

Regulatory agencies are going to required some sort of signed, dated, and controlled document (that means pretty much what you think it means — no one can change it after signing without some hoops to jump through and some paperwork) that states what the analysis is up front and how to determine whether a clinical trial meets its stated objectives. This is especially important when doing so-called “confirmatory trials” — the large ones that regulators rely on to decide whether a drug is effective.

Exploratory analyses are just that — ones designed to explore the data. If you get significant or “suggestive” results, you have to confirm them by conducting another study with a prospectively defined analysis, or else they are just conjectures.

Distinguishing between the two is very important for knowing what a study is telling us. Unfortunately, this kind of information is sometimes hard to get, and you rarely get it from a press release. Usually, the primary objective of a trial reported in an article has been prospectively defined, especially if the hypotheses were clearly stated. Sometimes, other information that is reported as part of a trial is exploratory, and should not be held with the same confidence (but rather with more curiosity). Subgroup analyses (i.e. primary analyses repeated on subsets of the population, say just males, or just one race, or just one age group) are often exploratory, unless the authors state up front that there was a scientific reason for looking.

When digging deeper into an issue addressed by a study, one of the things you uncover should be whether an analysis was stated before the study or after the data was collected.

%d bloggers like this: