Friday, February 10, 2012

Meta-Analyzing

Publication bias apparent here
The proliferation of meta-analyses over the last several years is astounding. When I collaborated on my first meta-analysis in 2004 (available here), the availability of resources to direct and assist the meta-analyst were substantially fewer than are available today.  On the Stata bookstore website, for example, you can currently find at least four books dedicated to the topic:  "Meta-Analysis in Stata:  An Updated Collection from the Stata Journal", "Systematic Reviews in Health Care:  Meta-Analysis in Context", "Introduction to Meta-Analysis", and "Methods for Meta-Analysis in Medical Research".  All four are terrific text books and impart the basics -- as well as some of the more esoteric and advanced material -- necessary for the successful conduct of a meta-analysis.  This isn't to say, however, that I'm an undisputed expert and that my first time wasn't, at times, a bit painful (when is it not?).  I remember receiving the responses from the reviewers after submitting the paper in 2004 and being overwhelmed by the lengthy and "what the hell does this mean?" nature of the comments pertaining to the statistical methods.  In retrospect, the comments were warranted since we were, after all, meta-analyzing nearly two dozen non-randomized, uncontrolled, cohort studies -- an approach that was somewhat at odds with the predominant view at the time that meta-analyses should only be conducted on randomized, controlled trials.  But we did it anyway and I wrote a wordy and cumbersome "Data Analysis" section that is almost painful to read now.  (To be fair, though, we also had to explain and justify why we assigned non-zero values to those effect sizes that were technically zero using a procedure known as "Windsorizing", thus adding another layer of complexity.)

But back to meta-analyzing, although if we really want to be proper, we'll take one step further back and start with systematic reviewing.  In some circles, "systematic review" and "meta-analysis" are used interchangeably although this is pushing the boundaries of acceptance.  To be precise, a systematic review is "a review that has been prepared using a systematic approach to minimizing biases and random errors which is documented in a materials and methods section" whereas a meta-analysis is a "statistical analysis of the results from independent studies, which generally aim to produce a single estimate of a treatment effect (Systematic Reviews in Health Care:  Meta-Analysis in Context, 2001).  A systematic review may or may not include a meta-analysis but a meta-analysis -- if done properly -- ought to be couched in a systematic review.  I suppose, then, that a meta-analysis is composed of two general phases:  first, study identification/data abstraction and, second, statistical synthesis of the individual trial results.

The first phase -- the identification, retrieval, and abstraction of the data from the studies comprising a meta-analysis -- must be systematic and reproducible.  In both of the meta-analyses I've worked on, my insanely productive friend & clinician was primarily responsible for this task. My responsibility, however, lie with the second phase:  data import & manipulation, estimation of overall effect, generation of graphs, identification of heterogeneity, and sensitivity analysis.  The data import, labeling, and manipulation is straightforward enough but the how's and why's of the statistical estimations can be simultaneously intuitive and confounding.  Trisha Greenhalgh regards this as "the statisticians' chance to pull a double whammy on you" by way of "frighten[ing] you will all the statistical tests in the individual papers, and then us[ing] a whole new battery of tests to produce a new set of odds ratios, confidence intervals, and values for significance" (How to Read a Paper:  The Basics of Evidence-Based Medicine, 2010).  This may be true, but all it really boils down to is just estimating an overall effect by combining the data.  In my first meta-analysis, the overall effect was a rate (a value bound between zero and one) whereas in the second meta-analysis, the overall effect was a relative risk.  In both papers, a "weighted" analysis was done in that the individual papers/trials with more subjects had more influence.  Once an overall effect was estimated, all the estimates from the individual trials as well as the overall effect were plotted on a "forest plot".  The forest plot from the second meta-analysis I worked on (available here) is posted below:


You can't mistake the forest for the trees with this plot
The Stata code to generate the overall effect and output the forest plot is, unexpectedly, rather short and straightforward (Stata's meta-analysis capabilities are, by far, the most advanced and extensive among the various statistical analysis software programs). 

metan ab_num_inf ab_num_noinf st_num_inf st_num_noinf, rr /// by(single_site) random label(namevar=author) counts /// 
lcols(author year) astext(55) textsize(140) ///
favours("Antibiotic-Impregnated Shunts" # "Standard Shunts") ///
nowarning xlabel(0.001, 0.458, 1)
Of course, things don't stop at the estimation of the overall effect.  Heterogeneity should be examined and if significant and substantial heterogeneity is observed then it's often prudent to explore the heterogeneity via sub-group analysis, sensitivity analysis, or even a meta-regression.  Another consideration is publication bias.  This bias follows from the tendency of only positive studies (both large and small) to get published and, consequently, to only be included in a meta-analysis.  One way to investigate this is with a "funnel plot" (pictured at the beginning of this post).  This graph is a scatter plot of the treatment effects from the individual studies on the x-axis and some measure of study size (e.g. standard error) on the y-axis.  If no publication bias is evident then a symmetric inverted funnel should be displayed.  If, however, some degree of asymmetry is evident then it is likely that publication bias may be present (smaller studies showing no significant effect are absent).  There was some evidence of publication bias in the second meta-analysis paper. 

Systematic reviewing and meta-analyzing are taking on larger and larger roles in medical research, especially since evidence-based medicine has become more necessary than ever.  I'm by no means an expert in meta-analysis -- and until I am -- I'll continue to access and rely on the many tools and texts available.

Thursday, February 9, 2012

Running Toward Injury?

There seems to be a spate of articles in the Health section of the NY Times lately about running and the most recent is no exception.  In the article, the author reports on a recent study published in "Medicine & Science in Sports and Exercise" by a group at Harvard that investigated the relationship between foot strike and injury rates in endurance runners.  Although the piece in the NY Times is informative enough, I wanted to read the actual article and judge their findings and conclusions for myself.  As most runners are well aware, the debate between heel-striking and forefoot-striking is thick with passionate opinion on both sides and with no real resolution expected anytime soon.  Like many runners, my interest in minimalist running was piqued by Christopher McDougall's "Born to Run" -- I was clearly enamored by it in my blog post -- and decided to follow up my enthusiasm with a slow transition to minimalist shoes.  Unfortunately, I think my enthusiasm may have been either premature or misplaced altogether since, insofar as I can tell, the onset of Achilles tendonitis in June 2011 may have followed, in part, from nine months of semi-regular minimalist running.  I'm obviously not 100% sure of the cause, nevertheless, my enthusiasm for minimalist shoes has tempered -- my Vibrams are languishing in the corner of my bedroom -- and I'm running all my road & trail miles (as modest as they are these days) in a pair of trail shoes, the Montrail Mountain Masochists.  Commentary aside, I downloaded the article from the journal (you are also welcome to access it by clicking here) to read what these researchers actually did and how they (statistically) did it. 

First, the authors cite the three primary strike patters among distance runners:  rear-foot strike (RFS), forefoot strike (FFS), and mid-foot strike (MFS).  With RFS, the heel contacts the ground first whereas with FFS, the ball of the foot strikes the ground first, and with MFS, the heel and ball of the foot contact the ground simultaneously.  The context in which their research question and hypothesis are introduced makes clear that their study was motivated by the increasing number of runners adopting FFS or MFS patterns and jettisoning their cushioned shoes because of claims that are often, at best, anecdotal and, at worse, unsubstantiated.  These authors aimed to add some rigor to the debate by way of investigating the differences in injury rates/types in a retrospective cohort of 52 Harvard cross country runners.  The authors hypothesized that certain injuries would associate with different strike types:  Achilles tendinopathies, foot pain, and stress fractures of metatarsals with FFS and hip pain, knee pain, lower back pain, tibial stress injuries, plantar fasciitis, and stress fractures of lower limbs (excluding metatarsals) with RFS.  The statistical analyses seem reasonable (t-tests for two-sample comparisons and a General Linear Model, GLM, for the multivariable model) although the numbers of tests run, sans a multiple test correction, gives me pause since they are analyzing four outcome variables -- repetitive injury rate, traumatic injury rate, FFS injury rate, and RFS injury rate -- and are comparing FFS vs RFS both by sex and overall.  This results in a lot of p-values, some of which may reach significance at an alpha level of 0.05 just by sheer chance.  That said, however, the overall rate of injury across the study was quite high:  nearly 75% of subjects suffered at least one moderate or severe repetitive stress injury per year.  When only repetitive injuries are only considered, the rates appear to be much higher for RFS than for FFS:  2.5 times higher when only mild & moderate injuries are considered and 1.7 times higher when moderate & severe combined.  There was no statistical difference between RFS and FFS for traumatic injury rates.  In short, the authors conclude that habitual RFS experience rates of repetitive injury are nearly twice that of habitual FFS (see table below, copy-and-pasted from the article).  Compelling evidence?  Sure, but not completely impenetrable.  


First, the sample size, although modest (N=52; n=16 FFS and n=36 RFS), should be larger (especially given the number of statistical comparisons made in the paper).  Second, the sample isn't generalizable:  a cohort of highly competitive, highly trained Ivy League runners isn't representative of the at large running population.  Third, it is unknown how and why the subjects adopted the strike pattern they did -- the lack of randomization makes it difficult to assert that the only discernible difference between the two groups is their foot strike.  Fourth, the senior author (Daniel E. Lieberman) acknowledges receiving funding from VibramUSA.  This doesn't automatically undermine his findings -- a fair amount of respectable medical research is underwritten by pharmaceutical companies -- although there is evidence in the literature to suggest that negative or null research is either left unpublished or is presented such that it better reflects the product of the funder.  (I'm in no way suggesting the senior author acted in such a way, just that it could.) 

All things considered, this paper is a solid contribution to the RFS vs. FFS debate.  Unlike a lot of retrospective studies that rely on subject recall -- a major limitation -- this one analyzed data that was collected contemporaneously (data collection and occurrence of event coincided) so report of mileage and injury are likely to be quite accurate.  

Lastly, the authors' comment about transitioning from RFS (cushioned shoes?) to FFS (minimalist shoes?) rang true (reproduced below).  If only I'd been this deliberate after "Born to Run". 

"Another point to consider is that this study did not test  for the effect of transitioning from RFS to FFS running, and it is unclear and unknown if runners who switch from RFS to FFS strikes will have lower injury rates. FFS running requires stronger calf muscles because eccentric or isometric contractions of the triceps surae are necessary to control ankle dorsiflexion at the beginning of stance, and shod FFS runners also generate higher joint moments in the ankle. Runners who transition to FFS running may be more likely to suffer from Achilles tendinopathies and calf muscle strains. FFS running also requires stronger foot muscles, so even though impact forces generated by FFS landings are low, runners who transition are perhaps more likely to experience forefoot pain or stress fractures. They may also experience plantar fasciitis if their foot muscles are weak. However, these injuries are treatable, and they may be preventable if runners transition, slowly, gradually, and with good overall form."