Table of Contents
Viechtbauer (2005)
The Methods and Data
In this paper, I compare the statistical properties of 5 different estimators for the amount of heterogeneity in the context of the random-effects model, including the Hunter-Schmidt (HS), Hedges (HE), DerSimonian-Laird (DL), maximum likelihood (ML), and restricted maximum likelihood (REML) estimator. Two examples are used to illustrate that the various estimators can provide divergent or even conflicting estimates. The first data set provides the results for $k=10$ studies that examined the effectiveness of open versus traditional education programs on student creativity. The second dataset provides the results for $k=18$ studies comparing open versus traditional education using student self-concept as the outcome variable. Both datasets were obtained from Hedges and Olkin (1985) and use the standardized mean difference as the effect size measure.
Example 1
The data for the first example can be created with:
dat <- data.frame( id = 1:10, yi = c(-0.581, 0.530, 0.771, 1.031, 0.553, 0.295, 0.078, 0.573, -0.176, -0.232), vi = c(0.023, 0.052, 0.060, 0.115, 0.095, 0.203, 0.200, 0.211, 0.051, 0.040))
After loading the metafor package with library(metafor)
, we can fit a random-effects model to these data using the various heterogeneity estimators with:
res.HS <- rma(yi, vi, data=dat, method="HS") res.HE <- rma(yi, vi, data=dat, method="HE") res.DL <- rma(yi, vi, data=dat, method="DL") res.ML <- rma(yi, vi, data=dat, method="ML") res.REML <- rma(yi, vi, data=dat, method="REML") res.EB <- rma(yi, vi, data=dat, method="EB") res.SJ <- rma(yi, vi, data=dat, method="SJ")
Note that the empirical Bayes (EB) and the Sidik-Jonkman (SJ) estimators are also included here for the sake of completeness. Next, we can combine the model objects into a list with:
res <- list(res.HS, res.HE, res.DL, res.ML, res.REML, res.EB, res.SJ)
And finally, we can extract the estimated values of $\tau^2$ with:
data.frame(method=sapply(res, function(x) x$method), tau2=sapply(res, function(x) round(x$tau2,3)), I2=sapply(res, function(x) round(x$I2,2)), H2=sapply(res, function(x) round(x$H2,2)))
which yields:
method tau2 I2 H2 1 HS 0.228 77.23 4.39 2 HE 0.148 68.80 3.21 3 DL 0.277 80.44 5.11 4 ML 0.197 74.51 3.92 5 REML 0.223 76.84 4.32 6 EB 0.192 74.05 3.85 7 SJ 0.199 74.75 3.96
The values above are the same as those given on page 271 in the paper. Also included in the output above are the estimated values of $I^2$ and $H^2$.
Example 2
The data for the second example can be created with:
dat <- data.frame( id = 1:18, yi = c(0.100, -0.162, -0.090, -0.049, -0.046, -0.010, -0.431, -0.261, 0.134, 0.019, 0.175, 0.056, 0.045, 0.103, 0.121, -0.482, 0.290, 0.342), vi = c(0.016, 0.015, 0.050, 0.050, 0.032, 0.052, 0.036, 0.024, 0.034, 0.033, 0.031, 0.034, 0.039, 0.167, 0.134, 0.096, 0.016, 0.035))
We can now repeat the same steps as above:
res.HS <- rma(yi, vi, data=dat, method="HS") res.HE <- rma(yi, vi, data=dat, method="HE") res.DL <- rma(yi, vi, data=dat, method="DL") res.ML <- rma(yi, vi, data=dat, method="ML") res.REML <- rma(yi, vi, data=dat, method="REML") res.EB <- rma(yi, vi, data=dat, method="EB") res.SJ <- rma(yi, vi, data=dat, method="SJ") res <- list(res.HS, res.HE, res.DL, res.ML, res.REML, res.EB, res.SJ) data.frame(method=sapply(res, function(x) x$method), tau2=sapply(res, function(x) round(x$tau2,3)), I2=sapply(res, function(x) round(x$I2,2)), H2=sapply(res, function(x) round(x$H2,2)))
yielding:
method tau2 I2 H2 1 HS 0.010 22.93 1.30 2 HE 0.000 0.00 1.00 3 DL 0.013 27.53 1.38 4 ML 0.013 28.45 1.40 5 REML 0.016 32.02 1.47 6 EB 0.010 23.72 1.31 7 SJ 0.025 42.67 1.74
The same values are given on page 272 in the paper. Especially the second example illustrates how conclusions about the presence of heterogeneity can depend on the estimator used. While the HE estimator suggests that the true effects are homogeneous, the REML and SJ estimators suggest more substantial amounts of heterogeneity (with corresponding $I^2$ values of 32% and 43%).
References
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.
Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model. Journal of Educational and Behavioral Statistics, 30(3), 261–293.