analyses:raudenbush2009
Differences
This shows you the differences between two versions of the page.
Next revision | |||
— | analyses:raudenbush2009 [2019/05/15 19:14] – external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ===== Raudenbush (2009) ===== | ||
+ | |||
+ | ==== The Methods and Data ==== | ||
+ | |||
+ | Raudenbush (2009) is an excellent chapter in //The handbook of research synthesis and meta-analysis// | ||
+ | |||
+ | Here, I will reproduce the results from Raudenbush (2009). The data are provided in Table 16.1 (p. 300) in the chapter and can be loaded with: | ||
+ | <code rsplus> | ||
+ | library(metafor) | ||
+ | dat <- dat.raudenbush1985 | ||
+ | dat | ||
+ | </ | ||
+ | (I copy the dataset into ' | ||
+ | <code output> | ||
+ | | ||
+ | 1 1 | ||
+ | 2 2 Conn et al. 1968 21 | ||
+ | 3 3 Jose & Cody 1971 19 | ||
+ | 4 4 | ||
+ | 5 5 | ||
+ | 6 6 Evans & Rosenthal 1968 | ||
+ | 7 7 | ||
+ | 8 8 | ||
+ | 9 9 | ||
+ | 10 10 Maxwell 1970 | ||
+ | 11 11 | ||
+ | 12 12 Flowers 1966 | ||
+ | 13 13 Keshock 1970 | ||
+ | 14 14 Henrikson 1970 | ||
+ | 15 15 Fine 1972 17 | ||
+ | 16 16 Grieger 1970 | ||
+ | 17 17 Rosenthal & Jacobson 1968 | ||
+ | 18 18 | ||
+ | 19 19 | ||
+ | </ | ||
+ | Note that there is a typo in Table 16.1 in the book chapter: The observed standardized mean difference for study 10 was $0.80$ and not $0.87$ (cf. Raudenbush and Bryk, 1985, Table 1). | ||
+ | |||
+ | ==== Fixed-Effects Model ==== | ||
+ | |||
+ | The results from a model assuming homogeneous effects can now be obtained with: | ||
+ | <code rsplus> | ||
+ | res.FE <- rma(yi, vi, data=dat, digits=3, method=" | ||
+ | res.FE | ||
+ | </ | ||
+ | <code output> | ||
+ | Fixed-Effects Model (k = 19) | ||
+ | |||
+ | Test for Heterogeneity: | ||
+ | Q(df = 18) = 35.830, p-val = 0.007 | ||
+ | |||
+ | Model Results: | ||
+ | |||
+ | estimate | ||
+ | | ||
+ | |||
+ | --- | ||
+ | Signif. codes: | ||
+ | </ | ||
+ | These results match exactly what is reported in Table 16.2 (p. 301). In particular, $\hat{\theta} = 0.060$, | ||
+ | |||
+ | **Sidenote**: | ||
+ | |||
+ | ==== Random-Effects Model ==== | ||
+ | |||
+ | Next, Raudenbush (2009) uses REML estimation to fit a random-effects model. Since REML estimation is the default for the '' | ||
+ | <code rsplus> | ||
+ | res.RE <- rma(yi, vi, data=dat, digits=3) | ||
+ | res.RE | ||
+ | </ | ||
+ | <code output> | ||
+ | Random-Effects Model (k = 19; tau^2 estimator: REML) | ||
+ | |||
+ | tau^2 (estimated amount of total heterogeneity): | ||
+ | tau (square root of estimated tau^2 value): | ||
+ | I^2 (total heterogeneity / total variability): | ||
+ | H^2 (total variability / sampling variability): | ||
+ | |||
+ | Test for Heterogeneity: | ||
+ | Q(df = 18) = 35.830, p-val = 0.007 | ||
+ | |||
+ | Model Results: | ||
+ | |||
+ | estimate | ||
+ | | ||
+ | |||
+ | --- | ||
+ | Signif. codes: | ||
+ | </ | ||
+ | Again, these are the same results reported in Table 16.2 (p. 301). In particular, $\hat{\tau}^2 = .019$, $\hat{\mu} = .084$, $SE[\hat{\mu}] = .052$, and $z = 1.62$ for the test $H_0: \mu = 0$. | ||
+ | |||
+ | ==== Credibility/ | ||
+ | |||
+ | Raudenbush (2009) also reports the results from a 95% credibility/ | ||
+ | <code rsplus> | ||
+ | predict(res.RE) | ||
+ | </ | ||
+ | <code output> | ||
+ | pred se ci.lb ci.ub cr.lb cr.ub | ||
+ | 0.089 0.056 -0.020 0.199 -0.245 0.423 | ||
+ | </ | ||
+ | but note that the interval ($-0.245$ to $0.423$) is a bit wider than the one reported in the book chapter (p. 302). That is because the credibility interval is computed with $\hat{\mu} \pm 1.96 \sqrt{\hat{\tau}^2 + SE[\hat{\mu}]^2}$ in the metafor package (see [[: | ||
+ | |||
+ | ==== Measures of Heterogeneity ==== | ||
+ | |||
+ | As additional measures of heterogeneity, | ||
+ | <code rsplus> | ||
+ | res.DL <- rma(yi, vi, data=dat, digits=3, method=" | ||
+ | res.DL | ||
+ | </ | ||
+ | <code output> | ||
+ | Random-Effects Model (k = 19; tau^2 estimator: DL) | ||
+ | |||
+ | tau^2 (estimated amount of total heterogeneity): | ||
+ | tau (square root of estimated tau^2 value): | ||
+ | I^2 (total heterogeneity / total variability): | ||
+ | H^2 (total variability / sampling variability): | ||
+ | |||
+ | Test for Heterogeneity: | ||
+ | Q(df = 18) = 35.830, p-val = 0.007 | ||
+ | |||
+ | Model Results: | ||
+ | |||
+ | estimate | ||
+ | | ||
+ | |||
+ | --- | ||
+ | Signif. codes: | ||
+ | </ | ||
+ | Now, $I^2 = 100 \times (35.83 - 18)/ 35.83 = 49.76%$ and $H^2 = 35.83 / 18 = 1.99$. So, if you want metafor to use the more conventional equations for computing $I^2$ and $H^2$, use '' | ||
+ | |||
+ | ==== Mixed-Effects Model ==== | ||
+ | |||
+ | Next, Raudenbush (2009) use a mixed-effects model with the number of prior contact weeks as predictor/ | ||
+ | <code rsplus> | ||
+ | dat$weeks.c <- ifelse(dat$weeks > 3, 3, dat$weeks) | ||
+ | </ | ||
+ | Then, the mixed-effects model can be fitted with: | ||
+ | <code rsplus> | ||
+ | res.ME <- rma(yi, vi, mods = ~ weeks.c, data=dat, digits=3) | ||
+ | res.ME | ||
+ | </ | ||
+ | <code output> | ||
+ | Mixed-Effects Model (k = 19; tau^2 estimator: REML) | ||
+ | |||
+ | tau^2 (estimated amount of residual heterogeneity): | ||
+ | tau (square root of estimated tau^2 value): | ||
+ | I^2 (residual heterogeneity / unaccounted variability): | ||
+ | H^2 (unaccounted variability / sampling variability): | ||
+ | R^2 (amount of heterogeneity accounted for): 100.00% | ||
+ | |||
+ | Test for Residual Heterogeneity: | ||
+ | QE(df = 17) = 16.571, p-val = 0.484 | ||
+ | |||
+ | Test of Moderators (coefficient(s) 2): | ||
+ | QM(df = 1) = 19.258, p-val < .001 | ||
+ | |||
+ | Model Results: | ||
+ | |||
+ | | ||
+ | intrcpt | ||
+ | weeks.c | ||
+ | |||
+ | --- | ||
+ | Signif. codes: | ||
+ | </ | ||
+ | These results again match the results in Table 16.2. The residual amount of heterogeneity is now $\hat{\tau}^2 \approx 0$ and the test statistic for $H_0: \tau^2 = 0$ is $Q_E(df=17) = 16.57$. The estimated model is $E(d_i) = .407 - 0.157 x_i$, where $x_i$ is the number of prior contact weeks. The standard errors of the model coefficients are $SE[b_0] = .087$ and $SE[b_1] = .036$, so that the test statistics are $z_0 = 4.68$ and $z_1 = -4.39$, respectively.((Note that there is a typo in Table 16.2. The test statistic for the slope is not -4.90, but -4.39.)) The amount of heterogeneity accounted for by the moderator included in the mixed-effects model (p. 304) is given under '' | ||
+ | |||
+ | ==== Empirical Bayes Estimates ==== | ||
+ | |||
+ | The unconditional shrinkage (empirical Bayes) estimates discussed on pages 304-305 can be obtained with: | ||
+ | <code rsplus> | ||
+ | blup(res.RE) | ||
+ | </ | ||
+ | <code output> | ||
+ | | ||
+ | 1 0.054 0.095 -0.132 0.241 | ||
+ | 2 0.101 0.104 -0.103 0.304 | ||
+ | 3 -0.006 0.110 -0.223 0.210 | ||
+ | 4 0.214 0.137 -0.053 0.482 | ||
+ | 5 0.105 0.136 -0.162 0.372 | ||
+ | 6 -0.008 0.084 -0.174 0.157 | ||
+ | 7 0.017 0.084 -0.148 0.183 | ||
+ | 8 -0.029 0.122 -0.269 0.210 | ||
+ | 9 0.160 0.110 -0.054 0.375 | ||
+ | 10 0.249 0.127 0.000 0.497 | ||
+ | 11 0.162 0.132 -0.097 0.421 | ||
+ | 12 0.110 0.123 -0.130 0.351 | ||
+ | 13 0.065 0.131 -0.192 0.321 | ||
+ | 14 0.110 0.131 -0.146 0.367 | ||
+ | 15 -0.029 0.108 -0.241 0.183 | ||
+ | 16 0.026 0.110 -0.191 0.242 | ||
+ | 17 0.191 0.101 -0.008 0.389 | ||
+ | 18 0.074 0.079 -0.081 0.230 | ||
+ | 19 0.025 0.112 -0.195 0.245 | ||
+ | </ | ||
+ | For example, for study 4, the empirical Bayes estimate is $\hat{\theta}_4 = 0.21$ (p. 305).((Note that equation 16.25 in Raudenbush (2009) is an approximation for the posterior variance of the empirical Bayes estimates. The metafor package uses the exact equation (see Appendix in Raudenbush & Bryk, 1985).)) The range of the empirical Bayes estimates is: | ||
+ | <code rsplus> | ||
+ | round(range(blup(res)$pred), | ||
+ | </ | ||
+ | <code output> | ||
+ | [1] -0.03 0.25 | ||
+ | </ | ||
+ | as reported in Table 16.2 and on page 305. | ||
+ | |||
+ | For the mixed-effects model, the conditional shrinkage estimates can be obtained with: | ||
+ | <code rsplus> | ||
+ | blup(res.ME) | ||
+ | </ | ||
+ | <code output> | ||
+ | | ||
+ | 1 0.093 0.037 0.020 0.166 | ||
+ | 2 -0.065 0.046 -0.155 0.026 | ||
+ | 3 -0.065 0.046 -0.155 0.026 | ||
+ | 4 0.407 0.087 0.237 0.578 | ||
+ | 5 0.407 0.087 0.237 0.578 | ||
+ | 6 -0.065 0.046 -0.155 0.026 | ||
+ | 7 -0.065 0.046 -0.155 0.026 | ||
+ | 8 -0.065 0.046 -0.155 0.026 | ||
+ | 9 0.407 0.087 0.237 0.578 | ||
+ | 10 0.250 0.057 0.139 0.361 | ||
+ | 11 0.407 0.087 0.237 0.578 | ||
+ | 12 0.407 0.087 0.237 0.578 | ||
+ | 13 0.250 0.057 0.139 0.361 | ||
+ | 14 0.093 0.037 0.020 0.166 | ||
+ | 15 -0.065 0.046 -0.155 0.026 | ||
+ | 16 -0.065 0.046 -0.155 0.026 | ||
+ | 17 0.250 0.057 0.139 0.361 | ||
+ | 18 0.093 0.037 0.020 0.166 | ||
+ | 19 -0.065 0.046 -0.155 0.026 | ||
+ | </ | ||
+ | These values are actually the same now as the fitted values (since $\hat{\tau}^2 \approx 0$), which can be obtained with (not shown): | ||
+ | <code rsplus> | ||
+ | predict(res.ME) | ||
+ | </ | ||
+ | |||
+ | ==== Alternative $\tau^2$ Estimators ==== | ||
+ | |||
+ | The results in Table 16.3 using the " | ||
+ | <code rsplus> | ||
+ | res.std <- list() | ||
+ | |||
+ | res.std$FE | ||
+ | res.std$ML | ||
+ | res.std$REML <- rma(yi, vi, data=dat, digits=3, method=" | ||
+ | res.std$DL | ||
+ | res.std$HE | ||
+ | |||
+ | round(t(sapply(res.std, | ||
+ | </ | ||
+ | <code output> | ||
+ | tau2 mu se | ||
+ | FE 0.000 0.060 0.036 1.655 -0.011 0.132 | ||
+ | ML 0.013 0.078 0.047 1.637 -0.015 0.171 | ||
+ | REML 0.019 0.084 0.052 1.621 -0.018 0.185 | ||
+ | DL 0.026 0.089 0.056 1.601 -0.020 0.199 | ||
+ | HE 0.080 0.114 0.079 1.443 -0.041 0.270 | ||
+ | </ | ||
+ | The results under " | ||
+ | |||
+ | ==== Knapp & Hartung Method ==== | ||
+ | |||
+ | The " | ||
+ | <code rsplus> | ||
+ | res.knha <- list() | ||
+ | |||
+ | res.knha$FE | ||
+ | res.knha$ML | ||
+ | res.knha$REML <- rma(yi, vi, data=dat, digits=3, method=" | ||
+ | res.knha$DL | ||
+ | res.knha$HE | ||
+ | |||
+ | round(t(sapply(res.knha, | ||
+ | </ | ||
+ | <code output> | ||
+ | tau2 mu se | ||
+ | FE 0.000 0.060 0.051 1.173 -0.048 0.168 | ||
+ | ML 0.013 0.078 0.059 1.311 -0.047 0.202 | ||
+ | REML 0.019 0.084 0.062 1.359 -0.046 0.213 | ||
+ | DL 0.026 0.089 0.064 1.405 -0.044 0.223 | ||
+ | HE 0.080 0.114 0.071 1.608 -0.035 0.264 | ||
+ | </ | ||
+ | |||
+ | ==== Huber-White Method ==== | ||
+ | |||
+ | The results using the Huber-White method (as described on pages 313-314) can be obtained with: | ||
+ | <code rsplus> | ||
+ | res.hw <- list() | ||
+ | |||
+ | res.hw$FE | ||
+ | res.hw$ML | ||
+ | res.hw$REML <- robust(res.std$REML, | ||
+ | res.hw$DL | ||
+ | res.hw$HE | ||
+ | |||
+ | round(t(sapply(res.hw, | ||
+ | </ | ||
+ | <code output> | ||
+ | tau2 mu se | ||
+ | FE 0.000 0.060 0.040 1.515 -0.023 0.144 | ||
+ | ML 0.013 0.078 0.047 1.637 -0.022 0.178 | ||
+ | REML 0.019 0.084 0.050 1.676 -0.021 0.189 | ||
+ | DL 0.026 0.089 0.052 1.710 -0.020 0.199 | ||
+ | HE 0.080 0.114 0.062 1.850 -0.015 0.244 | ||
+ | </ | ||
+ | Note that '' | ||
+ | |||
+ | ==== Final Note ==== | ||
+ | |||
+ | Please note that there appear to be a few typos in Table 16.3. Most of the smaller discrepancies are probably due to rounding differences, | ||
+ | |||
+ | ==== References ==== | ||
+ | |||
+ | Hedges, L. V. (1983). A random effects model for effect sizes. // | ||
+ | |||
+ | Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. // | ||
+ | |||
+ | Knapp, G., & Hartung, J. (2003). Improved tests for a random effects meta-regression with a single covariate. // | ||
+ | |||
+ | Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. //Journal of Educational Psychology, 76//(1), 85--97. | ||
+ | |||
+ | Raudenbush, S. W. (2009). Analyzing effect sizes: Random effects models. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), //The handbook of research synthesis and meta-analysis// | ||
+ | |||
+ | Raudenbush, S. W., & Bryk, A. S. (1985). Empirical Bayes meta-analysis. //Journal of Educational Statistics, 10//(2), 75--98. | ||
+ | |||
+ | Riley, R. D., Higgins, J. P., & Deeks, J. J. (2011). Interpretation of random effects meta-analyses. //British Medical Journal, 342//, d549. | ||
analyses/raudenbush2009.txt · Last modified: 2022/08/03 17:09 by Wolfgang Viechtbauer