tips:comp_two_independent_estimates
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
tips:comp_two_independent_estimates [2021/05/04 11:06] – Wolfgang Viechtbauer | tips:comp_two_independent_estimates [2023/02/18 19:24] – Wolfgang Viechtbauer | ||
---|---|---|---|
Line 6: | Line 6: | ||
We will use the ' | We will use the ' | ||
+ | |||
<code rsplus> | <code rsplus> | ||
library(metafor) | library(metafor) | ||
Line 12: | Line 13: | ||
dat | dat | ||
</ | </ | ||
+ | |||
<code output> | <code output> | ||
| | ||
Line 32: | Line 34: | ||
First, we fit two separate random-effects models within each subset defined by the '' | First, we fit two separate random-effects models within each subset defined by the '' | ||
+ | |||
<code rsplus> | <code rsplus> | ||
res1 <- rma(yi, vi, data=dat, subset=alloc==" | res1 <- rma(yi, vi, data=dat, subset=alloc==" | ||
Line 38: | Line 41: | ||
We then combine the estimates and standard errors from each model into a data frame. We also add a variable to distinguish the two models and, for reasons to be explained in more detail below, we add the estimated amounts of heterogeneity within each subset to the data frame. | We then combine the estimates and standard errors from each model into a data frame. We also add a variable to distinguish the two models and, for reasons to be explained in more detail below, we add the estimated amounts of heterogeneity within each subset to the data frame. | ||
+ | |||
<code rsplus> | <code rsplus> | ||
dat.comp <- data.frame(estimate = c(coef(res1), | dat.comp <- data.frame(estimate = c(coef(res1), | ||
Line 43: | Line 47: | ||
dat.comp | dat.comp | ||
</ | </ | ||
+ | |||
<code output> | <code output> | ||
estimate | estimate | ||
Line 49: | Line 54: | ||
</ | </ | ||
- | We can now compare the two estimates (i.e., the estimated average log risk ratios) by feeding them back to the '' | + | We can now compare the two estimates (i.e., the estimated average log risk ratios) by feeding them back to the '' |
<code rsplus> | <code rsplus> | ||
rma(estimate, | rma(estimate, | ||
</ | </ | ||
+ | |||
<code output> | <code output> | ||
Fixed-Effects with Moderators Model (k = 2) | Fixed-Effects with Moderators Model (k = 2) | ||
- | Test for Residual Heterogeneity: | + | Test for Residual Heterogeneity: |
QE(df = 0) = 0.000, p-val = 1.000 | QE(df = 0) = 0.000, p-val = 1.000 | ||
- | Test of Moderators (coefficient(s) 2): | + | Test of Moderators (coefficient(s) 2): |
QM(df = 1) = 1.946, p-val = 0.163 | QM(df = 1) = 1.946, p-val = 0.163 | ||
Model Results: | Model Results: | ||
- | estimate | + | estimate |
intrcpt | intrcpt | ||
- | metarandom | + | metarandom |
--- | --- | ||
Signif. codes: | Signif. codes: | ||
</ | </ | ||
+ | |||
While we find that studies using random assignment obtain on average larger (i.e., more negative) effects than studies not using random assignment ($b_1 = -0.490$, $SE = 0.351$), the difference between the two estimates is not significant ($z = -1.395$, $p = .163$). | While we find that studies using random assignment obtain on average larger (i.e., more negative) effects than studies not using random assignment ($b_1 = -0.490$, $SE = 0.351$), the difference between the two estimates is not significant ($z = -1.395$, $p = .163$). | ||
The test of the difference between the two estimates is really just a [[https:// | The test of the difference between the two estimates is really just a [[https:// | ||
+ | |||
<code rsplus> | <code rsplus> | ||
with(dat.comp, | with(dat.comp, | ||
</ | </ | ||
+ | |||
<code output> | <code output> | ||
- | zval | + | zval |
-1.395 | -1.395 | ||
</ | </ | ||
+ | |||
This is the same value that we obtained above. | This is the same value that we obtained above. | ||
Line 86: | Line 97: | ||
Now let's take a different approach, fitting a meta-regression model with '' | Now let's take a different approach, fitting a meta-regression model with '' | ||
+ | |||
<code rsplus> | <code rsplus> | ||
rma(yi, vi, mods = ~ alloc, data=dat, digits=3) | rma(yi, vi, mods = ~ alloc, data=dat, digits=3) | ||
</ | </ | ||
+ | |||
<code output> | <code output> | ||
Mixed-Effects Model (k = 13; tau^2 estimator: REML) | Mixed-Effects Model (k = 13; tau^2 estimator: REML) | ||
Line 98: | Line 111: | ||
R^2 (amount of heterogeneity accounted for): 0.00% | R^2 (amount of heterogeneity accounted for): 0.00% | ||
- | Test for Residual Heterogeneity: | + | Test for Residual Heterogeneity: |
QE(df = 11) = 138.511, p-val < .001 | QE(df = 11) = 138.511, p-val < .001 | ||
- | Test of Moderators (coefficient(s) 2): | + | Test of Moderators (coefficient(s) 2): |
QM(df = 1) = 1.833, p-val = 0.176 | QM(df = 1) = 1.833, p-val = 0.176 | ||
Model Results: | Model Results: | ||
- | | + | |
intrcpt | intrcpt | ||
- | allocrandom | + | allocrandom |
--- | --- | ||
Signif. codes: | Signif. codes: | ||
</ | </ | ||
+ | |||
The result is very similar to what we saw earlier: The coefficient corresponding to the '' | The result is very similar to what we saw earlier: The coefficient corresponding to the '' | ||
Line 120: | Line 134: | ||
Using the '' | Using the '' | ||
+ | |||
<code rsplus> | <code rsplus> | ||
rma.mv(yi, vi, mods = ~ alloc, random = ~ alloc | trial, struct=" | rma.mv(yi, vi, mods = ~ alloc, random = ~ alloc | trial, struct=" | ||
</ | </ | ||
+ | |||
<code output> | <code output> | ||
Multivariate Meta-Analysis Model (k = 13; method: REML) | Multivariate Meta-Analysis Model (k = 13; method: REML) | ||
- | Variance Components: | + | Variance Components: |
outer factor: trial (nlvls = 13) | outer factor: trial (nlvls = 13) | ||
Line 135: | Line 151: | ||
tau^2.2 | tau^2.2 | ||
- | Test for Residual Heterogeneity: | + | Test for Residual Heterogeneity: |
QE(df = 11) = 138.511, p-val < .001 | QE(df = 11) = 138.511, p-val < .001 | ||
- | Test of Moderators (coefficient(s) 2): | + | Test of Moderators (coefficient(s) 2): |
QM(df = 1) = 1.946, p-val = 0.163 | QM(df = 1) = 1.946, p-val = 0.163 | ||
Model Results: | Model Results: | ||
- | | + | |
intrcpt | intrcpt | ||
- | allocrandom | + | allocrandom |
--- | --- | ||
Signif. codes: | Signif. codes: | ||
</ | </ | ||
- | Note that the two estimates of $\tau^2$ are now identical to the ones we obtained earlier from the separate random-effects models. Also, the coefficient, | ||
- | A discussion/ | + | Note that the two estimates of $\tau^2$ are now identical to the ones we obtained earlier from the separate random-effects models. Also, the coefficient, |
+ | |||
+ | A discussion/ | ||
+ | |||
+ | Rubio-Aparicio, | ||
Rubio-Aparicio, | Rubio-Aparicio, | ||
- | We can also do a likelihood ratio test (LRT) to examine whether there are significant differences in the $\tau^2$ values across subsets. This can be done with: | + | We can also conduct |
<code rsplus> | <code rsplus> | ||
Line 163: | Line 182: | ||
anova(res1, res0) | anova(res1, res0) | ||
</ | </ | ||
+ | |||
<code output> | <code output> | ||
- | df | + | df |
- | Full 4 29.2959 30.8875 35.9626 -10.6480 | + | Full 4 29.2959 30.8875 35.9626 -10.6480 |
Reduced | Reduced | ||
</ | </ | ||
So in this example, we would not reject the null hypothesis $H_0: \tau^2_1 = \tau^2_2$ ($p = .58$). | So in this example, we would not reject the null hypothesis $H_0: \tau^2_1 = \tau^2_2$ ($p = .58$). | ||
+ | |||
+ | ==== Other Types of Models ==== | ||
+ | |||
+ | The issue discussed above also arises for other types of models (e.g., multilevel meta-analytic models). When fitting a particular model within several subgroups, then the variance components of the model are automatically allowed to differ across the subgroups. On the other hand, when fitting the same type of model to all studies combined (but including a moderator to allow the mean effect size to differ across subgroups), then the variance components are assumed to be the same within each subgroups (unless one takes extra steps as illustrated above to allow the variance components to differ across subgroups). Consequently, | ||
+ |
tips/comp_two_independent_estimates.txt · Last modified: 2024/06/18 19:28 by Wolfgang Viechtbauer