tips:model_selection_with_glmulti_and_mumin
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionLast revisionBoth sides next revision | ||
tips:model_selection_with_glmulti_and_mumin [2022/08/03 12:51] – Wolfgang Viechtbauer | tips:model_selection_with_glmulti_and_mumin [2022/09/25 10:45] – Wolfgang Viechtbauer | ||
---|---|---|---|
Line 1: | Line 1: | ||
===== Model Selection using the glmulti and MuMIn Packages ===== | ===== Model Selection using the glmulti and MuMIn Packages ===== | ||
- | Information-theoretic approaches provide methods for model selection and (multi)model inference that differ quite a bit from more traditional methods based on null hypothesis testing (e.g., Anderson, | + | Information-theoretic approaches provide methods for model selection and (multi)model inference that differ quite a bit from more traditional methods based on null hypothesis testing (e.g., Anderson, |
==== Data Preparation ==== | ==== Data Preparation ==== | ||
- | For the example, I will use data from the meta-analysis by Bangert-Drowns et al. (2004) on the effectiveness of school-based writing-to-learn interventions on academic achievement ('' | + | For the example, I will use data from the meta-analysis by Bangert-Drowns et al. (2004) on the effectiveness of school-based writing-to-learn interventions on academic achievement ('' |
<code rsplus> | <code rsplus> | ||
library(metafor) | library(metafor) | ||
Line 43: | Line 43: | ||
* length: treatment length (in weeks) | * length: treatment length (in weeks) | ||
- | * wic: writing in class (0 = no; 1 = yes) | + | * wic: writing |
- | * feedback: feedback (0 = no; 1 = yes) | + | * feedback: feedback |
* info: writing contained informational components (0 = no; 1 = yes) | * info: writing contained informational components (0 = no; 1 = yes) | ||
* pers: writing contained personal components (0 = no; 1 = yes) | * pers: writing contained personal components (0 = no; 1 = yes) | ||
* imag: writing contained imaginative components (0 = no; 1 = yes) | * imag: writing contained imaginative components (0 = no; 1 = yes) | ||
- | * meta: prompts for metacognitive reflection (0 = no; 1 = yes) | + | * meta: prompts for metacognitive reflection |
More details about the meaning of these variables can be found in Bangert-Drowns et al. (2004). For the purposes of this illustration, | More details about the meaning of these variables can be found in Bangert-Drowns et al. (2004). For the purposes of this illustration, | ||
Line 59: | Line 59: | ||
==== Model Selection ==== | ==== Model Selection ==== | ||
- | We will now examine the fit and plausibility of various models, focusing on models that contain none, one, and up to seven (i.e., all) of these moderator variables. For this, we need to install and load the glmulti package and define a function that takes a model formula and a dataset as input and then fits a random/mixed-effects meta-regression model to the given data using maximum likelihood estimation: | + | We will now examine the fit and plausibility of various models, focusing on models that contain none, one, and up to seven (i.e., all) of these moderator variables. For this, we install and load the glmulti package and define a function that (a) takes a model formula and dataset as input and (b) then fits a mixed-effects meta-regression model to the given data using maximum likelihood estimation: |
<code rsplus> | <code rsplus> | ||
install.packages(" | install.packages(" | ||
Line 101: | Line 101: | ||
{{ tips: | {{ tips: | ||
- | The horizontal red line differentiates between models whose AICc value is less versus more than 2 units away from that of the " | + | The horizontal red line differentiates between models whose AICc value is less versus more than 2 units away from that of the " |
But let's take a look at the top 10 models: | But let's take a look at the top 10 models: | ||
Line 123: | Line 123: | ||
</ | </ | ||
- | We see that the " | + | We see that the " |
So, we could now examine the " | So, we could now examine the " | ||
Line 178: | Line 178: | ||
Now we can carry out the computations for multimodel inference with: | Now we can carry out the computations for multimodel inference with: | ||
<code rsplus> | <code rsplus> | ||
- | coef(res) | + | coef(res, varweighting=" |
</ | </ | ||
The output is not shown, because I don't find it very intuitive. But with a bit of extra code, we can make it more interpretable: | The output is not shown, because I don't find it very intuitive. But with a bit of extra code, we can make it more interpretable: | ||
<code rsplus> | <code rsplus> | ||
- | mmi <- as.data.frame(coef(res)) | + | mmi <- as.data.frame(coef(res, varweighting=" |
mmi <- data.frame(Estimate=mmi$Est, | mmi <- data.frame(Estimate=mmi$Est, | ||
mmi$z <- mmi$Estimate / mmi$SE | mmi$z <- mmi$Estimate / mmi$SE | ||
Line 195: | Line 195: | ||
<code output> | <code output> | ||
| | ||
- | intrcpt | + | intrcpt |
- | imag | + | imag |
- | meta | + | meta |
- | feedback | + | feedback |
- | length | + | length |
- | pers | + | pers |
- | wic | + | wic |
- | info -0.0183 | + | info -0.0183 |
</ | </ | ||
- | I rounded the results to 4 digits to make the results easier to interpret. Note that the table again includes the importance values. In addition, we get unconditional estimates of the model coefficients (first column). These are model-averaged parameter estimates, which are weighted averages of the model coefficients across the various models (with weights equal to the model probabilities). These values are called " | + | I rounded the results to 4 digits to make the results easier to interpret. Note that the table again includes the importance values. In addition, we get unconditional estimates of the model coefficients (first column). These are model-averaged parameter estimates, which are weighted averages of the model coefficients across the various models (with weights equal to the model probabilities). These values are called " |
==== Multimodel Predictions ==== | ==== Multimodel Predictions ==== | ||
Line 280: | Line 280: | ||
So, the candidate set would include over $2 \times 10^8$ possible models. Fitting all of these models would not only test our patience (and would be a waste of valuable CPU cycles), it would also be a pointless exercise (even fitting the 128 models above could be critiqued by some as a mindless hunting expedition -- although if one does not get too fixated on //the// best model, but considers all the models in the set as part of a multimodel inference approach, this critique loses some of its force). So, I won't consider this any further in this example. | So, the candidate set would include over $2 \times 10^8$ possible models. Fitting all of these models would not only test our patience (and would be a waste of valuable CPU cycles), it would also be a pointless exercise (even fitting the 128 models above could be critiqued by some as a mindless hunting expedition -- although if one does not get too fixated on //the// best model, but considers all the models in the set as part of a multimodel inference approach, this critique loses some of its force). So, I won't consider this any further in this example. | ||
- | |||
- | ==== Other Model Types ==== | ||
- | |||
- | The same principle can of course be applied when fitting other types of models, such as those that can be fitted with the '' | ||
- | |||
- | For multivariate/ | ||
==== Using the MuMIn Package ==== | ==== Using the MuMIn Package ==== | ||
Line 331: | Line 325: | ||
Multimodel inference can be done with: | Multimodel inference can be done with: | ||
<code rsplus> | <code rsplus> | ||
- | summary(model.avg(res, revised.var=FALSE)) | + | summary(model.avg(res)) |
</ | </ | ||
<code output> | <code output> | ||
Line 337: | Line 331: | ||
(full average) | (full average) | ||
Estimate Std. Error z value Pr(>|z|) | Estimate Std. Error z value Pr(>|z|) | ||
- | intrcpt | + | intrcpt |
- | imag 0.351153 | + | imag 0.351153 |
- | meta 0.051201 | + | meta 0.051201 |
- | feedback | + | feedback |
- | length | + | length |
- | wic -0.017004 | + | wic -0.017004 |
- | pers 0.013244 | + | pers 0.013244 |
- | info | + | info |
</ | </ | ||
- | I have removed some of the output, since this is the part we are most interested in. These are the same results as in object '' | + | I have removed some of the output, since this is the part we are most interested in. These are the same results as in object '' |
Finally, the relative importance values for the predictors can be obtained with: | Finally, the relative importance values for the predictors can be obtained with: | ||
<code rsplus> | <code rsplus> | ||
- | importance(res) | + | sw(res) |
</ | </ | ||
<code output> | <code output> | ||
Line 358: | Line 352: | ||
</ | </ | ||
These are again the same values we obtained earlier. | These are again the same values we obtained earlier. | ||
+ | |||
+ | ==== Other Model Types ==== | ||
+ | |||
+ | The same principle can of course be applied when fitting other types of models, such as those that can be fitted with the '' | ||
+ | |||
+ | For multivariate/ | ||
==== References ==== | ==== References ==== | ||
- | Anderson, D. R. (2007). //Model based inference in the life sciences: A primer on evidence//. New York: Springer. | + | Anderson, D. R. (2008). //Model based inference in the life sciences: A primer on evidence//. New York: Springer. |
Bangert-Drowns, | Bangert-Drowns, |
tips/model_selection_with_glmulti_and_mumin.txt · Last modified: 2022/10/13 06:07 by Wolfgang Viechtbauer