Wednesday, August 24, 2011

"I think you'll find it's a bit more complicated than that"


I just finished reading Ben Goldacre's "Bad Science:  Quacks, Hacks, and Big Pharma Flacks".  I liked it.  Actually, I really liked it.  What I liked most about it, though, was how you could sense how pissed off Goldacre is without the book devolving into a tired and lengthy tirade.  Goldacre is a medical doctor and, perhaps more importantly, a scientist that values data and data-driven evidence above all else.  He has no patience for anecdote, sham studies, and pseudo-scientific professions, namely homeopaths and nutritionists.  And he rails against the mass media and their inability (unwillingness?) to accurately report science stories resulting in a further "dumbing down" of the media and, most disturbingly, of the masses. 

I hesitate to reduce the book to a single sentence -- a theme -- but if forced to, a sentence on page 52 would suffice:  "Transparency and detail are everything in science."  So, so true.  But what of the execution of this mandate?  The use and presentation of statistics is a natural place to focus one's efforts, although the process starts long before the graphs are generated and the p-values are calculated:  "Overall, doing research robustly and fairly...simply requires that you think before you start" (pp. 53).  As for the statistics, Goldacre discusses statistical sleights of hand employed by mainstream medicine as well as devotes an entire chapter ("Bad Stats") to discuss how statistics are misused and misunderstood.  In his discussion on how mainstream medicine -- big pharma -- tries to distort data and results, he proffers the following tricks used by the pharmaceutical industry:

1.  Study the drug/device in winners.  That is, avoid enrolling people on lots of drugs or with lots of complications -- the "no-hopers" -- since they are less likely than a younger, healthier subject to exhibit any improvement. 
2.  Compare the drug against a useless control.  In this scenario, the drug companies intentionally avoid comparing their drug to the standard treatment (another drug on market) since placebo-controlled trials are more likely to yield unambiguously positive results.
3.  When comparing to another drug, administer the competing drug incorrectly or in a different dose.  This trick seems especially underhanded since only those with an intimate knowledge of the drug being manipulated would be able to identify the manipulation.  Nevertheless, doing this is intended to increase the incidence of side effects of the competing drug and, thus, make the experimental drug look much better.
4.  Avoid collecting data about side effects or collect them in such a way that key side effects are down-played or ignored altogether.
5.  Employ a "surrogate outcome"  rather than a real-world outcome.  For example, designate reduced cholesterol rather than cardiac death as the endpoint.  Reaching this endpoint is easier, cheaper, and faster to achieve.
6.  Bury the disappointing or negative data in your trial in the text of the paper.  And certainly don't highlight it by graphing it or mentioning it in the abstract or conclusion.  
7.  Don't publish or postpone publishing negative data after a long delay.  Perhaps in the meantime, another study returning less negative results (maybe even positive results?) -- and one likely conducted by the same investigators -- might supersede the negative results?  (Although Goldacre doesn't mention it in this section -- he does elsewhere -- sometimes the non-publication of negative results isn't altogether the investigators fault:  virtually all journals have been shown to exhibit "publication bias":  the bias towards publishing only studies with positive, statistically significant results.) 

And the more statistical-related tricks:

8.  Ignore the protocol entirely.  Rather than stick to the statistical analysis plan outlined in the protocol, run as many statistical tests as possible and report any statistically significant association, especially if it supports (even tangentially) your thesis.
9.  Play with the baseline.  "Adjust" for baseline values depending on which group is doing better at the beginning of the study:  adjust if the placebo group is better off and don't adjust if the treatment group is better off.
10.  Ignore dropouts:  These folks tend to fare worse in trials so don't aggressively follow them up for the endpoint or include them in analysis.
11.  Clean up the data:  Selectively include or exclude data based on how the data points affect the drug's performance.
12.  The best of...:  This refers to use of flexible interim analysis, that is, premature stopping of the trial if the results statistically favor the experimental drug or extension of the trial by a few months in hopes that the nearly significant results become significant.
13.  Torture the data.  Conduct all sorts of sub-group analyses in hopes that something, anything, pops out as statistically significant.
14.  Try every button on the computer:  Run every statistical test you can think of irrespective of whether it is appropriate or not.  You're bound to hit on something significant!

In the "Bad Stats" chapter, Goldacre briefly discusses the idea of "statistically significant" and since this is a central feature of research, it's worth repeating Goldacre's definition here:  "'statistically significant' [is] just a way of expressing the likelihood that the result you got was attributable merely to chance" (pp. 194).  He goes on:  "The standard cutoff point for statistical significance is a p-value of 0.05, which is just another way of saying, 'If I did this experiment a hundred times, I'd expect a spurious positive result on five occasions, just by chance'" (pp. 194-5).  

Science and data-driven decision making are, unfortunately, under full-on assault in the world (especially in the United States?) thus making this book as timely and necessary as ever.

No comments:

Post a Comment