Differences

This shows you the differences between two versions of the page.

--- tips:comp_two_independent_estimates [2023/02/18 17:35] – Wolfgang Viechtbauer
+++ tips:comp_two_independent_estimates [2024/03/28 09:01] – Wolfgang Viechtbauer
@@ Line 43: / Line 43: @@
 <code rsplus>
-dat.comp <- data.frame(estimate = c(coef(res1), coef(res2)), stderror = c(res1$se, res2$se),
+dat.comp <- data.frame(meta     = c("random", "other"),
-                       meta = c("random","other"), tau2 = round(c(res1$tau2, res2$tau2),3))
+                       estimate = c(coef(res1), coef(res2)),
-dat.comp
+                       stderror = c(res1$se, res2$se),
+                       tau2     = c(res1$tau2, res2$tau2))
+dfround(dat.comp, 3)
 </code>
 <code output>
-    estimate  stderror   meta  tau2
+    meta estimate stderror  tau2
--0.9709645 0.2759557 random 0.393
+ random   -0.971    0.276 0.393
--0.4812706 0.2169886  other 0.212
+  other   -0.481    0.217 0.212
 </code>
@@ Line 79: / Line 81: @@
 </code>
-While we find that studies using random assignment obtain on average larger (i.e., more negative) effects than studies not using random assignment ($b_1 = -0.490$, $SE = 0.351$), the difference between the two estimates is not significant ($z = -1.395$, $p = .163$).
+While we find that studies using random assignment obtain on average larger (i.e., more negative) effects than studies not using random assignment ($b_1 = -0.490$, $SE = 0.351$), the difference between the two estimates is not statistically significant ($z = -1.395$, $p = .163$).
 The test of the difference between the two estimates is really just a [[https://en.wikipedia.org/wiki/Wald_test|Wald-type test]], given by the equation $$z = \frac{\hat{\mu}_1 - \hat{\mu}_2}{\sqrt{SE[\hat{\mu}_1]^2 + SE[\hat{\mu}_2]^2}},$$ where $\hat{\mu}_1$ and $\hat{\mu}_2$ are the two estimates and $SE[\hat{\mu}_1]$ and $SE[\hat{\mu}_2]$ the corresponding standard errors. The test statistics can therefore also be computed with:
@@ Line 129: / Line 131: @@
 The result is very similar to what we saw earlier: The coefficient corresponding to the ''alloc'' dummy is equal to $b_1 = -0.490$ ($SE = 0.362$) and not significant ($p = .176$).
-However, the results are not exactly identical. The reason for this is as follows. When we fit separate random-effects models in the two subsets, we are allowing the amount of heterogeneity within each set to be different (as shown earlier, the estimates were $\hat{\tau}^2 = 0.393$ and $\hat{\tau}^2 = 0.212$ for studies using and not using random assignment, respectively). On the other hand, the mixed-effects meta-regression model fitted above has a single variance component for the amount of residual heterogeneity, which implies that the amount of heterogeneity //within each subset// is assumed to be the same ($\hat{\tau}^2 = 0.318$ in this example).
+However, the results are not exactly identical. The reason for this is as follows. When we fit separate random-effects models in the two subsets, we are allowing the amount of heterogeneity within each set to be different (as shown earlier, the estimates were $\hat{\tau}^2 = 0.393$ and $\hat{\tau}^2 = 0.212$ for studies using and not using random assignment, respectively). On the other hand, the mixed-effects meta-regression model fitted above has a single variance component for the amount of residual heterogeneity, which implies that the amount of heterogeneity //within each subset// is assumed to be the same ($\hat{\tau}^2 = 0.318$ in this example -- note that this falls somewhere between the two $\hat{\tau}^2$ values we obtained within the subsets).
 ==== Meta-Regression with All Studies but Different Amounts of (Residual) Heterogeneity ====
@@ Line 136: / Line 138: @@
 <code rsplus>
-rma.mv(yi, vi, mods = ~ alloc, random = ~ alloc | trial, struct="DIAG", data=dat, digits=3)
+rma.mv(yi, vi, mods = ~ alloc, random = ~ alloc | trial,
+       struct="DIAG", data=dat, digits=3)
 </code>
@@ Line 169: / Line 172: @@
 Note that the two estimates of $\tau^2$ are now identical to the ones we obtained earlier from the separate random-effects models. Also, the coefficient, standard error, and p-value for the moderator now matches the results obtained earlier.((See also [[tips:different_tau2_across_subgroups|here]] for another example to illustrate various approaches for allowing $\tau^2$ to differ across subgroups.))
-A discussion/comparison of these two approaches (i.e., assuming a single $\tau^2$ value or allowing $\tau^2$ to differ across subsets) can be found in the following article:
+A discussion/comparison of these two approaches (i.e., assuming a single $\tau^2$ value or allowing $\tau^2$ to differ across subsets) can be found in the following articles:
+Rubio-Aparicio, M., Sánchez-Meca, J., López-López, J. A., Botella, J. & Marín-Martínez, F. (2017). Analysis of categorical moderators in mixed-effects meta-analysis: Consequences of using pooled versus separate estimates of the residual between-studies variances. //British Journal of Mathematical and Statistical Psychology, 70//(3), 439-456. [[https://doi.org/https://doi.org/10.1111/bmsp.12092]]
 Rubio-Aparicio, M., López-López, J. A., Viechtbauer, W., Marín-Martínez, F., Botella, J., & Sánchez-Meca, J. (2020). Testing categorical moderators in mixed-effects meta-analysis in the presence of heteroscedasticity. //Journal of Experimental Education, 88//(2), 288-310. [[https://doi.org/10.1080/00220973.2018.1561404]]
-We can also do a likelihood ratio test (LRT) to examine whether there are significant differences in the $\tau^2$ values across subsets. This can be done with:
+We can also conduct a likelihood ratio test (LRT) to examine whether there are significant differences in the $\tau^2$ values across subsets. This can be done with:
 <code rsplus>
@@ Line 188: / Line 193: @@
 So in this example, we would not reject the null hypothesis $H_0: \tau^2_1 = \tau^2_2$ ($p = .58$).
+==== Other Types of Models ====
+The issue discussed above also arises for other types of models (e.g., multilevel meta-analytic models). When fitting a particular model within several subgroups, then the variance components of the model are automatically allowed to differ across the subgroups. On the other hand, when fitting the same type of model to all studies combined (but including a moderator to allow the mean effect size to differ across subgroups), then the variance components are assumed to be the same within each subgroups (unless one takes extra steps as illustrated above to allow the variance components to differ across subgroups). Consequently, the results obtained with these two approaches will not be identical.