The metafor Package

A Meta-Analysis Package for R

User Tools

Site Tools


tips:comp_mh_different_software

Differences

This shows you the differences between two versions of the page.


Last revision
tips:comp_mh_different_software [2019/05/05 16:21] – external edit 127.0.0.1
Line 1: Line 1:
 +===== Comparison of the Mantel-Haenszel Method in Different Software =====
 +
 +The Mantel-Haenszel method is an approach for fitting meta-analytic fixed-effects models when dealing with studies providing data in the form of 2x2 tables or in the form of event counts (i.e., person-time data) for two groups (Mantel & Haenszel, 1959). The method is particularly advantageous when aggregating a large number of studies with small sample sizes (the so-called sparse data or increasing strata case).
 +
 +The method is available in the metafor package via the ''rma.mh()'' function. By default, the results obtained may differ slightly from those obtained via the ''metan'' function in Stata (for more details, see Harris et al., 2008; Sterne, 2009), the [[http://tech.cochrane.org/Revman|Review Manager]] (RevMan) from the [[http://www.cochrane.org/|Cochrane Collaboration]], or [[http://www.meta-analysis.com/|Comprehensive Meta-Analysis]] (CMA). The reason for such discrepancies is explained further below using an illustrative dataset from a meta-analysis comparing the risk of catheter-related bloodstream infection (CRBSI) when using anti-infective-treated versus standard catheters in the acute care setting (Niel-Weise et al., 2007).
 +
 +==== Data Preparation ====
 +
 +The data to be used for this example are stored in the dataset ''dat.nielweise2007'':
 +<code rsplus>
 +library(metafor)
 +dat <- dat.nielweise2007
 +dat
 +</code>
 +<code output>
 +   study         author year ai n1i ci n2i
 +1      1           Bach 1996  0 116  3 117
 +2      2         George 1997  1  44  3  35
 +3      3           Maki 1997  2 208  9 195
 +4      4           Raad 1997  0 130  7 136
 +5      5          Heard 1998  5 151  6 157
 +6      6         Collin 1999  1  98  4 139
 +7      7         Hannan 1999  1 174  3 177
 +8      8          Marik 1999  1  74  2  39
 +9      9         Pierce 2000  1  97 19 103
 +10    10          Sheng 2000  1 113  2 122
 +11    11 Chatzinikolaou 2003  0  66  7  64
 +12    12         Corral 2003  0  70  1  58
 +13    13   Brun-Buisson 2004  3 188  5 175
 +14    14           Leon 2004  6 187 11 180
 +15    15          Yucel 2004  0 118  0 105
 +16    16        Moretti 2005  0 252  1 262
 +17    17           Rupp 2005  1 345  3 362
 +18    18           Osma 2006  4  64  1  69
 +</code>
 +Variables ''ai'' and ''ci'' indicate the number of CRBSIs in patients receiving an anti-infective or a standard catheter, respectively, while ''n1i'' and ''n2i'' indicate the total number of patients in the respective groups. Note that the number of infections was quite low in many studies, with zero cases observed in several of the treatment groups. Also, no cases (infections) were observed in either group in the Yucel (2004) study.
 +
 +==== Mantel-Haenszel Method ====
 +
 +An analysis of these data using the Mantel-Haenszel method can be carried out with:
 +<code rsplus>
 +res1 <- rma.mh(measure="OR", ai=ai, n1i=n1i, ci=ci, n2i=n2i, data=dat)
 +print(res1, digits=3)
 +</code>
 +<code output>
 +Fixed-Effects Model (k = 18)
 +
 +Test for Heterogeneity:
 +Q(df = 16) = 16.864, p-val = 0.394
 +
 +Model Results (log scale):
 +
 +estimate       se     zval     pval    ci.lb    ci.ub
 +  -1.209    0.222   -5.434    <.001   -1.645   -0.773
 +
 +Model Results (OR scale):
 +
 +estimate    ci.lb    ci.ub
 +   0.299    0.193    0.462
 +
 +Cochran-Mantel-Haenszel Test:    CMH = 32.214, df = 1,  p-val < .001
 +Tarone's Test for Heterogeneity: X^2 = 25.679, df = 16, p-val = 0.059
 +</code>
 +Therefore, the odds ratio is estimated to be .299 (with 95% CI: 0.193 to 0.462). In other words, the odds of an infection are estimated to be approximately 70% lower (i.e., $(1 - .299) \times 100%$) in patients receiving an anti-infective-treated catheter instead of a standard catheter. The overall effect is clearly statistically significant (with both the Wald-type z-test and the Cochran-Mantel-Haenszel chi-square test in close agreement). The Q-test for heterogeneity is not significant ($Q(16) = 16.86, p = .39$), although Tarone's test is suggestive of potential heterogeneity.
 +
 +==== Results from Stata ====
 +
 +The same analysis run in Stata using the ''metan'' command (with default settings) yields the following results:
 +<code output>
 +           Study         OR    [95% Conf. Interval]     % Weight
 +---------------------+---------------------------------------------------
 +Bach (1996)          |  0.140       0.007     2.749          3.93
 +Brun-Buisson (2004)  |  0.551       0.130     2.342          5.77
 +Chatzinikolaou (2003 |  0.058       0.003     1.031          8.56
 +Collin (1999)        |  0.348       0.038     3.162          3.71
 +Corral (2003)        |  0.272       0.011     6.801          1.84
 +George (1997)        |  0.248       0.025     2.497          3.70
 +Hannan (1999)        |  0.335       0.035     3.255          3.35
 +Heard (1998)          0.862       0.257     2.886          6.44
 +Leon (2004)          |  0.509       0.184     1.408         12.29
 +Maki (1997)          |  0.201       0.043     0.941         10.42
 +Marik (1999)          0.253       0.022     2.887          2.93
 +Moretti (2005)        0.345       0.014     8.514          1.66
 +Osma (2006)          |  4.533       0.493    41.683          1.02
 +Pierce (2000)        |  0.046       0.006     0.351         20.65
 +Raad (1997)          |  0.066       0.004     1.170          8.27
 +Rupp (2005)          |  0.348       0.036     3.360          3.31
 +Sheng (2000)          0.536       0.048     5.990          2.16
 +Yucel (2004)          (Excluded)
 +---------------------+---------------------------------------------------
 +M-H pooled OR        |  0.317       0.208     0.483        100.00
 +---------------------+---------------------------------------------------
 +
 +  Heterogeneity chi-squared =  16.41 (d.f. = 16) p = 0.425
 +  I-squared (variation in OR attributable to heterogeneity) =   2.5%
 +
 +  Test of OR=1 : z=   5.36 p = 0.000
 +</code>
 +The following forest plot is also generated:
 +
 +{{ tips:comp_mh_forest_stata.png?nolink }}
 +
 +Note that the estimated overall odds ratio (and corresponding CI) is slightly different than the one obtained earlier. Also, the z-test of the overall effect and the chi-square test for heterogeneity are slightly different.
 +
 +==== Results from RevMan ====
 +
 +After entering the same data into the Review Manager and running the analogous analysis yields the following results:
 +
 +{{ tips:comp_mh_forest_revman.png?nolink&650 }}
 +
 +These results match what is reported by Stata and are again slightly different compared to the results obtained with metafor.
 +
 +==== Results from CMA ====
 +
 +Finally, the figure below shows the results from Comprehensive Meta-Analysis (CMA). These results match those obtained with Stata and RevMan and differ slightly from those obtained with metafor.
 +
 +{{ tips:comp_mh_forest_cma.png?nolink }}
 +
 +==== Reason for the Difference ====
 +
 +The results differ because studies with zero cases in either group are handled by default in a different way in metafor compared to Stata, RevMan, and CMA. To understand this better, note that the Mantel-Haenszel method itself does not require the calculation of the observed outcomes of the individual studies (in the present example, the observed (log) odds ratios of the $k$ studies) and instead directly makes use of the 2×2 table counts. Zero cells are not a problem (except in some extreme cases, such as when there are zero cases in one or both groups across all of the 2×2 tables). Therefore, it is unnecessary to add some constant to the cell counts of a study with zero cases in either group. However, both Stata, RevMan, and CMA apply an adjustment (often called a continuity correction) to the cell counts in such studies (but studies with zero cases in both groups are dropped/excluded from the method). In particular, ''1/2'' is added to each of the cells of the 2×2 table in such studies before applying the Mantel-Haenszel method.
 +
 +By default, metafor uses this adjustment when calculating the observed outcomes (the observed log odds ratios) of the $k$ studies (here, zero cells can be problematic, so adding a constant value to the cell counts ensures that all $k$ values can be calculated). Also, similarly, studies with zero cases in both groups are automatically dropped/excluded. However, when applying the Mantel-Haenszel method, no adjustment to the cell counts is made, since this is not necessary (and in fact can increase the bias in the Mantel-Haenszel method -- see Bradburn et al., 2007).
 +
 +We can, however, adjust the settings, so that metafor also applies the cell count adjustment, not only when calculating the observed outcomes, but also when carrying out the computations of the Mantel-Haenszel method. For this, we have to adjust the defaults of the ''add'', ''to'', and ''drop00'' arguments (see the documentation of the ''escalc()'' and ''rma.mh()'' functions for further details). In particular, we could use:
 +<code rsplus>
 +res2 <- rma.mh(measure="OR", ai=ai, n1i=n1i, ci=ci, n2i=n2i, data=dat, add=c(1/2,1/2), to=c("only0","only0"), drop00=c(TRUE,TRUE))
 +print(res2, digits=3)
 +</code>
 +<code output>
 +Fixed-Effects Model (k = 17)
 +
 +Test for Heterogeneity: 
 +Q(df = 16) = 16.406, p-val = 0.425
 +
 +Model Results (log scale):
 +
 +estimate       se     zval     pval    ci.lb    ci.ub 
 +  -1.149    0.214   -5.356    <.001   -1.569   -0.728 
 +
 +Model Results (OR scale):
 +
 +estimate    ci.lb    ci.ub 
 +   0.317    0.208    0.483 
 +
 +Cochran-Mantel-Haenszel Test:    CMH = 30.919, df = 1,  p-val < .001
 +Tarone's Test for Heterogeneity: X^2 = 22.033, df = 16, p-val = 0.142
 +</code>
 +
 +These are the exact same results as obtained with Stata, RevMan, and CMA. However, the results of Bradburn et al. (2007) suggest that the ''1/2'' adjustment should only be used with caution when applying the Mantel-Haenszel method. Also, alternative correction factors could be considered, which may actually lead to more accurate results (see Sweeting et al., 2004). Finally, the findings by Bradburn et al. (2007) suggest that Peto's method (as implemented in the ''rma.peto()'' function) can actually give the least biased results and may be preferrable when events are rare (as long as treatment and control groups are of approximately equal size within trials and the true odds ratio underlying the studies is not very large).
 +
 +==== References ====
 +
 +Bradburn, M. J., Deeks, J. J., Berlin, J. A., & Localio, A. R. (2007). Much ado about nothing: A comparison of the performance of meta-analytical methods with rare events. //Statistics in Medicine, 26//(1), 53--77.
 +
 +Harris, R. J., Bradburn, M. J., Deeks, J. J., Harbord, R. M., Altman, D. G., & Sterne, J. A. C. (2008). metan: Fixed- and random-effects meta-analysis. //The Stata Journal, 8//(1), 3--28. URL: http://www.stata-journal.com/article.html?article=sbe24_2
 +
 +Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. //Journal of the National Cancer Institute, 22//(4), 719--748.
 +
 +Niel-Weise, B. S., Stijnen, T., & van den Broek, P. J. (2007). Anti-infective-treated central venous catheters: A systematic review of randomized controlled trials. //Intensive Care Medicine, 33//(12), 2058--2068.
 +
 +Review Manager (RevMan) [Computer program] (Version 5.3). Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2012. URL: http://tech.cochrane.org/Revman
 +
 +Sterne, J. A. C. (Ed.) (2009). //Meta-analysis in Stata: An updated collection from the Stata Journal//. Stata Press, College Station, TX.
 +
 +Sweeting, M. J., Sutton, A. J., & Lambert, P. C. (2004). What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. //Statistics in Medicine, 23//(9), 1351--1375.
  
tips/comp_mh_different_software.txt · Last modified: 2021/11/08 15:48 by Wolfgang Viechtbauer