The metafor Package

A Meta-Analysis Package for R

tips:ci_for_r2

Confidence Intervals for $R^2$ in Meta-Regression Models

The (pseudo) $R^2$ statistic that is shown in the output for meta-regression models fitted with the rma() function provides an estimate of how much of the total amount of heterogeneity is accounted for by the moderator(s) included in the model (Raudenbush, 2009). An explanation of how this statistic is computed is provided here.

Based on a simulation study, we know that the $R^2$ statistic is not very accurate unless the number of studies ($k$) is quite large (López-López et al., 2014). To gauge the precision of the value of $R^2$ in a particular meta-analysis, we may want to compute a confidence interval (CI) for this statistic. However, this is difficult, because the $R^2$ statistic is based on the ratio of two heterogeneity estimates, one from the random-effects model (to obtain an estimate of the total amount of heterogeneity) and the other from the mixed-effects meta-regression model (to obtain an estimate of the residual amount of heterogeneity) that need to be fitted in turn. A possible approach we can take to construct a CI is to use bootstrapping. This is illustrated below (see also here for an illustration of using bootstrapping in the context of the random-effects model).

Data Preparation / Inspection

For this example, we will use the data from the meta-analysis by Bangert-Drowns et al. (2004) on the effectiveness of writing-to-learn interventions on academic achievement. For the purposes of this example, we will keep only the columns needed and only the rows where the moderators to be used in the meta-regression model below are completely coded (this results in two rows being removed from the dataset; note that these two rows would be removed anyway when fitting the model, but for bootstrapping, it is better to remove these rows manually beforehand).

dat <- dat.bangertdrowns2004
dat <- dat[c(1:3,9:12,15:16)]
dat <- dat[complete.cases(dat),]
head(dat)
  id   author year info pers imag meta     yi    vi
1  1 Ashworth 1992    1    1    0    1  0.650 0.070
2  2    Ayers 1993    1    1    1    0 -0.750 0.126
3  3   Baisch 1990    1    1    0    1 -0.210 0.042
4  4    Baker 1994    1    0    0    0 -0.040 0.019
5  5   Bauman 1992    1    1    0    1  0.230 0.022
6  6   Becker 1996    0    1    0    0  0.030 0.009
7  .        .    .    .    .    .    .      .     .

Variable yi provides the standardized mean differences and vi the corresponding sampling variances. Variables info, pers, imag, and meta are dummy variables to indicate whether the writing-to-learn intervention contained informational components, personal components, imaginative components, and prompts for metacognitive reflection, respectively (for the specific meaning of these variables, see Bangert-Drowns et al., 2004). We will use these 4 dummy variables as potential moderators.

Meta-Regression Model

We start by fitting the meta-regression model containing these 4 moderators as follows:

res <- rma(yi, vi, mods = ~ info + pers + imag + meta, data=dat)
res
Mixed-Effects Model (k = 46; tau^2 estimator: REML)

tau^2 (estimated amount of residual heterogeneity):     0.0412 (SE = 0.0192)
tau (square root of estimated tau^2 value):             0.2030
I^2 (residual heterogeneity / unaccounted variability): 51.71%
H^2 (unaccounted variability / sampling variability):   2.07
R^2 (amount of heterogeneity accounted for):            18.34%

Test for Residual Heterogeneity:
QE(df = 41) = 82.7977, p-val = 0.0001

Test of Moderators (coefficients 2:5):
QM(df = 4) = 10.0061, p-val = 0.0403

Model Results:

estimate      se     zval    pval    ci.lb   ci.ub
intrcpt    0.3440  0.2179   1.5784  0.1145  -0.0831  0.7711
info      -0.1988  0.2106  -0.9438  0.3453  -0.6115  0.2140
pers      -0.3223  0.1782  -1.8094  0.0704  -0.6715  0.0268   .
imag       0.2540  0.2023   1.2557  0.2092  -0.1424  0.6504
meta       0.4817  0.1678   2.8708  0.0041   0.1528  0.8106  **

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The omnibus test indicates that we can reject the null hypothesis that the 4 coefficients corresponding to these moderators are all simultaneously equal to zero ($Q_M(\textrm{df} = 4) = 10.01, p = .04$). Moreover, it appears that writing-to-learn interventions tend to be more effective on average when they include prompts for metacognitive reflection ($p = .0041$, with a coefficient that indicates that the size of the effect is on average 0.48 points higher when such prompts are given). Finally, the results indicate that around $R^2 = 18.3\%$ of the heterogeneity is accounted for by this model.

Bootstrap Confidence Interval

We will now use (non-parametric) bootstrapping to obtain a CI for $R^2$. For this, we load the boot package and write a short function with three inputs: the first provides the original data, the second is a vector of indices which define the bootstrap sample, and the third is the formula for the model to be fitted. The function then fits the same meta-regression model as above, but based on the bootstrap sample, and returns the value of the $R^2$ statistic. The purpose of the try() function is to catch cases where the model fitting algorithm does not converge.

library(boot)

boot.func <- function(dat, indices, formula) {
res <- try(suppressWarnings(rma(yi, vi, mods = formula, data=dat[indices,])), silent=TRUE)