# The metafor Package

A Meta-Analysis Package for R

### Sidebar

analyses:morris2008

## Morris (2008)

### The Purpose and Data

Morris (2008) discusses various ways for computing a (standardized) effect size measure for pretest posttest control group designs, where the characteristic, response, or dependent variable assessed in the individual studies is a quantitative variable.

As described by Becker (1988), we can compute the standardized mean change (with raw score standardization) for a treatment and control group with $$g_T = c(n_T-1) \frac{\bar{x}_{post,T} - \bar{x}_{pre,T}}{SD_{pre,T}}$$ and $$g_C = c(n_C-1) \frac{\bar{x}_{post,C} - \bar{x}_{pre,C}}{SD_{pre,C}},$$ where $\bar{x}_{pre,T}$ and $\bar{x}_{post,T}$ are the treatment group pretest and posttest means, $SD_{pre,T}$ is the standard deviation of the pretest scores, $c(m) = \sqrt{2/m} \Gamma[m/2] / \Gamma[(m-1)/2]$ is a bias-correction factor1), $n_T$ is the size of the treatment group, and $\bar{x}_{pre,C}$, $\bar{x}_{post,C}$, $SD_{pre,C}$, and $n_C$ are the analogous values for the control group. Then the difference in the two standardized mean change values, namely $$g = g_T - g_C$$ indicates how much larger (or smaller) the change in the treatment group was (in standard deviation units) when compared to the change in the control group. Values of $g$ computed for a number of studies could then be meta-analyzed with standard methods.2)

Morris (2008) uses five studies from a meta-analysis on training effectiveness by Carlson and Schmidt (1999) to illustrate these computations. We can create the same dataset with:

datT <- data.frame(
m_pre   = c(30.6, 23.5, 0.5, 53.4, 35.6),
m_post  = c(38.5, 26.8, 0.7, 75.9, 36.0),
sd_pre  = c(15.0, 3.1, 0.1, 14.5, 4.7),
sd_post = c(11.6, 4.1, 0.1, 4.4, 4.6),
ni      = c(20, 50, 9, 10, 14),
ri      = c(0.47, 0.64, 0.77, 0.89, 0.44))

and

datC <- data.frame(
m_pre   = c(23.1, 24.9, 0.6, 55.7, 34.8),
m_post  = c(19.7, 25.3, 0.6, 60.7, 33.4),
sd_pre  = c(13.8, 4.1, 0.2, 17.3, 3.1),
sd_post = c(14.8, 3.3, 0.2, 17.9, 6.9),
ni      = c(20, 42, 9, 11, 14),
ri      = c(0.47, 0.64, 0.77, 0.89, 0.44))

The contents of datT and datC are then:

  m_pre m_post sd_pre sd_post ni   ri
1  30.6   38.5   15.0    11.6 20 0.47
2  23.5   26.8    3.1     4.1 50 0.64
3   0.5    0.7    0.1     0.1  9 0.77
4  53.4   75.9   14.5     4.4 10 0.89
5  35.6   36.0    4.7     4.6 14 0.44

and

  m_pre m_post sd_pre sd_post ni   ri
1  23.1   19.7   13.8    14.8 20 0.47
2  24.9   25.3    4.1     3.3 42 0.64
3   0.6    0.6    0.2     0.2  9 0.77
4  55.7   60.7   17.3    17.9 11 0.89
5  34.8   33.4    3.1     6.9 14 0.44

### Computing Standardized Mean Change for Each Group

After loading the metafor package with library(metafor), the standardized mean change within each group can be computed with:

datT <- escalc(measure="SMCR", m1i=m_post, m2i=m_pre, sd1i=sd_pre, ni=ni, ri=ri, data=datT)
datC <- escalc(measure="SMCR", m1i=m_post, m2i=m_pre, sd1i=sd_pre, ni=ni, ri=ri, data=datC)

Now the contents of datT and datC are:

  m_pre m_post sd_pre sd_post ni   ri     yi     vi
1  30.6   38.5   15.0    11.6 20 0.47 0.5056 0.0594
2  23.5   26.8    3.1     4.1 50 0.64 1.0481 0.0254
3   0.5    0.7    0.1     0.1  9 0.77 1.8054 0.2322
4  53.4   75.9   14.5     4.4 10 0.89 1.4181 0.1225
5  35.6   36.0    4.7     4.6 14 0.44 0.0801 0.0802

and

  m_pre m_post sd_pre sd_post ni   ri      yi     vi
1  23.1   19.7   13.8    14.8 20 0.47 -0.2365 0.0544
2  24.9   25.3    4.1     3.3 42 0.64  0.0958 0.0173
3   0.6    0.6    0.2     0.2  9 0.77  0.0000 0.0511
4  55.7   60.7   17.3    17.9 11 0.89  0.2667 0.0232
5  34.8   33.4    3.1     6.9 14 0.44 -0.4250 0.0864

The standardized mean change values are given in the yi columns. Note that internally, the escalc() function computes m1i-m2i, so the argument m1i should be set equal to the posttest means and m2i to the pretest means if one wants to compute the standardized mean change in the way described above. The sampling variances (the values in the vi columns) are computed based on equation 13 in Becker (1988).

### Computing the Difference in the Standardized Mean Change

We can now compute the difference between the two standardized mean changes values for each study. In addition, since the treatment and control groups are independent, the corresponding sampling variances can be computed by adding up the sampling variances of the two groups:

dat <- data.frame(yi = datT$yi - datC$yi, vi = datT$vi + datC$vi)
round(dat, 2)
    yi   vi
1 0.74 0.11
2 0.95 0.04
3 1.81 0.28
4 1.15 0.15
5 0.51 0.17

The yi values above are the exact same value given in Table 5 (under the $d_{ppc1}$ column) by Morris (2008).

### Estimating the Sampling Variances

Equation 16 in Morris (2008) is the exact sampling variance of $g$. To actually compute the sampling variance in practice, the unknown parameters in this equation must be replaced with their sample counterparts. As noted earlier, the escalc() function actually uses a slightly different method to estimate the sampling variance (based on equation 13 in Becker, 1988). Hence, the values above and the ones given in Table 5 (column $\hat{\sigma}^2(d_{ppc1})$ in Morris, 2008) differ slightly.

There are in fact dozens of ways of how the sampling variance for the standardized mean change can be estimated (see Viechtbauer, 2008, Tables 2 and 3 – and even that is not an exhaustive list). Hence, there are dozens of ways of estimating the sampling variance of $g$ above. Differences should only be relevant in small samples.

### The Actual Meta-Analysis

For the actual meta-analysis part, we simply pass the yi and vi values to the rma() function. For example, a fixed-effects model can be fitted with:

rma(yi, vi, data=dat, method="FE", digits=2)
Fixed-Effects Model (k = 5)

Test for Heterogeneity:
Q(df = 4) = 4.43, p-val = 0.35

Model Results:

estimate       se     zval     pval    ci.lb    ci.ub
0.95     0.14     6.62     <.01     0.67     1.23      ***

---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Note that these results are slightly different than the ones in Table 5 due to the different ways of estimating the sampling variances.

### Alternative Estimates

In his article, Morris (2008) discusses two other ways of computing an effect size measure for pretest posttest control group designs. The second approach that pools the two pretest SDs actually can be more efficient under certain conditions. However, that approach assumes that the true pretest SDs are equal for the two groups. That may not be the case. The approach given above does not make that assumption and therefore is more broadly applicable (but may be less efficient).

If you really want to use the approach with pooled pretest SDs, then this can be done as follows:

sd_pool <- sqrt((with(datT, (ni-1)*sd_pre^2) + with(datC, (ni-1)*sd_pre^2)) / (datT$ni + datC$ni - 2))
dat <- data.frame(yi = metafor:::.cmicalc(datT$ni + datC$ni - 2) * (with(datT, m_post - m_pre) - with(datC, m_post - m_pre)) / sd_pool)