Friday, February 10, 2012


Publication bias apparent here
The proliferation of meta-analyses over the last several years is astounding. When I collaborated on my first meta-analysis in 2004 (available here), the availability of resources to direct and assist the meta-analyst were substantially fewer than are available today.  On the Stata bookstore website, for example, you can currently find at least four books dedicated to the topic:  "Meta-Analysis in Stata:  An Updated Collection from the Stata Journal", "Systematic Reviews in Health Care:  Meta-Analysis in Context", "Introduction to Meta-Analysis", and "Methods for Meta-Analysis in Medical Research".  All four are terrific text books and impart the basics -- as well as some of the more esoteric and advanced material -- necessary for the successful conduct of a meta-analysis.  This isn't to say, however, that I'm an undisputed expert and that my first time wasn't, at times, a bit painful (when is it not?).  I remember receiving the responses from the reviewers after submitting the paper in 2004 and being overwhelmed by the lengthy and "what the hell does this mean?" nature of the comments pertaining to the statistical methods.  In retrospect, the comments were warranted since we were, after all, meta-analyzing nearly two dozen non-randomized, uncontrolled, cohort studies -- an approach that was somewhat at odds with the predominant view at the time that meta-analyses should only be conducted on randomized, controlled trials.  But we did it anyway and I wrote a wordy and cumbersome "Data Analysis" section that is almost painful to read now.  (To be fair, though, we also had to explain and justify why we assigned non-zero values to those effect sizes that were technically zero using a procedure known as "Windsorizing", thus adding another layer of complexity.)

But back to meta-analyzing, although if we really want to be proper, we'll take one step further back and start with systematic reviewing.  In some circles, "systematic review" and "meta-analysis" are used interchangeably although this is pushing the boundaries of acceptance.  To be precise, a systematic review is "a review that has been prepared using a systematic approach to minimizing biases and random errors which is documented in a materials and methods section" whereas a meta-analysis is a "statistical analysis of the results from independent studies, which generally aim to produce a single estimate of a treatment effect (Systematic Reviews in Health Care:  Meta-Analysis in Context, 2001).  A systematic review may or may not include a meta-analysis but a meta-analysis -- if done properly -- ought to be couched in a systematic review.  I suppose, then, that a meta-analysis is composed of two general phases:  first, study identification/data abstraction and, second, statistical synthesis of the individual trial results.

The first phase -- the identification, retrieval, and abstraction of the data from the studies comprising a meta-analysis -- must be systematic and reproducible.  In both of the meta-analyses I've worked on, my insanely productive friend & clinician was primarily responsible for this task. My responsibility, however, lie with the second phase:  data import & manipulation, estimation of overall effect, generation of graphs, identification of heterogeneity, and sensitivity analysis.  The data import, labeling, and manipulation is straightforward enough but the how's and why's of the statistical estimations can be simultaneously intuitive and confounding.  Trisha Greenhalgh regards this as "the statisticians' chance to pull a double whammy on you" by way of "frighten[ing] you will all the statistical tests in the individual papers, and then us[ing] a whole new battery of tests to produce a new set of odds ratios, confidence intervals, and values for significance" (How to Read a Paper:  The Basics of Evidence-Based Medicine, 2010).  This may be true, but all it really boils down to is just estimating an overall effect by combining the data.  In my first meta-analysis, the overall effect was a rate (a value bound between zero and one) whereas in the second meta-analysis, the overall effect was a relative risk.  In both papers, a "weighted" analysis was done in that the individual papers/trials with more subjects had more influence.  Once an overall effect was estimated, all the estimates from the individual trials as well as the overall effect were plotted on a "forest plot".  The forest plot from the second meta-analysis I worked on (available here) is posted below:

You can't mistake the forest for the trees with this plot
The Stata code to generate the overall effect and output the forest plot is, unexpectedly, rather short and straightforward (Stata's meta-analysis capabilities are, by far, the most advanced and extensive among the various statistical analysis software programs). 

metan ab_num_inf ab_num_noinf st_num_inf st_num_noinf, rr /// by(single_site) random label(namevar=author) counts /// 
lcols(author year) astext(55) textsize(140) ///
favours("Antibiotic-Impregnated Shunts" # "Standard Shunts") ///
nowarning xlabel(0.001, 0.458, 1)
Of course, things don't stop at the estimation of the overall effect.  Heterogeneity should be examined and if significant and substantial heterogeneity is observed then it's often prudent to explore the heterogeneity via sub-group analysis, sensitivity analysis, or even a meta-regression.  Another consideration is publication bias.  This bias follows from the tendency of only positive studies (both large and small) to get published and, consequently, to only be included in a meta-analysis.  One way to investigate this is with a "funnel plot" (pictured at the beginning of this post).  This graph is a scatter plot of the treatment effects from the individual studies on the x-axis and some measure of study size (e.g. standard error) on the y-axis.  If no publication bias is evident then a symmetric inverted funnel should be displayed.  If, however, some degree of asymmetry is evident then it is likely that publication bias may be present (smaller studies showing no significant effect are absent).  There was some evidence of publication bias in the second meta-analysis paper. 

Systematic reviewing and meta-analyzing are taking on larger and larger roles in medical research, especially since evidence-based medicine has become more necessary than ever.  I'm by no means an expert in meta-analysis -- and until I am -- I'll continue to access and rely on the many tools and texts available.

No comments:

Post a Comment