tips:hunter_schmidt_method

The meta-analytic methods developed by Hunter and Schmidt (1990, 2004, 2014), sometimes called "psychometric meta-analysis", are commonly used to conduct meta-analyses in industrial/organizational psychology and related areas. A question that comes up on a regular basis is how one can conduct such meta-analyses using the metafor package. Adding such functionality is on my to-do list (see here), but it is already possible to conduct analyses in the style of the Hunter and Schmidt method already now. Note that there are some subtle differences between the H&S method and the statistical/theoretical framework underlying the `rma()`

function, so I would call this a H&S-type analysis – it is not an exact replication of the H&S method (and the corresponding software).

To conduct a basic meta-analysis of correlation coefficients using a H&S-type approach (what Hunter and Schmidt call a "bare-bones meta-analysis"), three adjustments are necessary. In particular, we need to:

- use an adjusted method for calculating the sampling variances of the correlation coefficients,
- use the sample sizes as weights in the analysis, and
- switch to the Hunter and Schmidt estimator to estimate the amount of heterogeneity.

**A few details about step 1**: The large-sample equation for the sampling variance of a (Pearson product-moment) correlation coefficient is $$\text{Var}[r_i] = \frac{(1-\rho_i^2)^2}{n_i - 1}$$ (although one could also just divide by $n_i$, and not $n_i - 1$). Since this equation involves the (unknown) true correlation coefficient $\rho_i$, it is not usable in practice. One approach is to substitute $r_i$ for $\rho_i$, but the resulting estimator for $\text{Var}[r_i]$ can be very inaccurate, especially when the sample size of a study is small. Hunter and Schmidt instead suggest to calculate the sample-size weighted average of the correlation coefficients, that is, $$\bar{r} = \frac{\sum n_i r_i}{\sum n_i},$$ and then substitute $\bar{r}$ into the equation for the variance. This makes perfect sense if the true correlations were actually homogeneous (since $\bar{r}$ is then a pretty good estimator for the true correlation across all studies). However, even if the true correlations are heterogeneous, this approach still tends to work rather well. The `escalc()`

function actually provides this functionality, by using the `vtype="AV"`

option (note: at the moment, you will have to install the development version of the metafor package for this to work; see here for instructions).

Let's use the data from the meta-analysis by McDaniel et al. (1990) (on the validity of employment interviews) for such an analysis. The three adjustments described above are applied as follows:

library(metafor) dat <- dat.mcdaniel1994 dat <- escalc(measure="COR", ri=ri, ni=ni, data=dat, vtype="AV") res <- rma(yi, vi, weights=ni, data=dat, method="HS") res

Random-Effects Model (k = 160; tau^2 estimator: HS) tau^2 (estimated amount of total heterogeneity): 0.0179 (SE = 0.0052) tau (square root of estimated tau^2 value): 0.1338 I^2 (total heterogeneity / total variability): 74.51% H^2 (total variability / sampling variability): 3.92 Test for Heterogeneity: Q(df = 159) = 647.6782, p-val < .0001 Model Results: estimate se zval pval ci.lb ci.ub 0.2005 0.0295 6.7947 <.0001 0.1427 0.2583 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

These results are very similar to what is reported in the article.

A key element of the Hunter and Schmidt method is the application of various "artifact corrections". For example, we know that due to measurement error in one or both variables of interest, the observed correlation coefficients will tend to be attenuated (i.e., lowered/dilluted). However, if we know the measurement reliabilities of the variables, one can easily apply a correction for attenuation. Unfortunately, the dataset does not contain those reliabilities, but we can illustrate the principle by just generating some reasonable measurement reliability values. For job performance, we will draw values from a uniform distribution with bounds .60 and .85 and for the interview ratings, we will draw values from a uniform distribution with bounds .65 and .90. Then we apply the correction for attenuation to the observed correlations and we also need to correct the sampling variances accordingly:

set.seed(513131) # just to make results fully replicable dat$relx <- round(runif(nrow(dat), 0.60, 0.85), 2) dat$rely <- round(runif(nrow(dat), 0.65, 0.90), 2) dat$yi.c <- dat$yi / sqrt(dat$relx * dat$rely) dat$vi.c <- dat$vi / (dat$relx * dat$rely)

Now we can proceed with the analysis as before, but using the corrected correlation coefficients and corresponding sampling variances. Also, Hunter and Schmidt recommend giving more weight not only to studies with larger sample sizes, but also studies that have higher reliability (or more generally, where the artifact correction factor is closer to 1). In essence, this translates into using a custom weighting scheme that uses the inverse of the corrected sampling variances. Hence, we use:

res <- rma(yi.c, vi.c, weights=1/vi.c, data=dat, method="HS") res

Random-Effects Model (k = 160; tau^2 estimator: HS) tau^2 (estimated amount of total heterogeneity): 0.0317 (SE = 0.0092) tau (square root of estimated tau^2 value): 0.1780 I^2 (total heterogeneity / total variability): 74.36% H^2 (total variability / sampling variability): 3.90 Test for Heterogeneity: Q(df = 159) = 643.4014, p-val < .0001 Model Results: estimate se zval pval ci.lb ci.ub 0.2680 0.0391 6.8624 <.0001 0.1915 0.3446 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Other types of corrections (e.g., for range restriction in one or both variables of interest) are described by Hunter and Schmidt and can be applied accordingly.

There is also the psychmeta package, which is specifically geared towards psychometric meta-analyses and which incorporates these types of corrections.

Hunter, J. E., & Schmidt, F. L. (1990). *Methods of meta-analysis: Correcting error and bias in research findings.* Newbury Park, CA: Sage.

Hunter, J. E., & Schmidt, F. L. (2004). *Methods of meta-analysis: Correcting error and bias in research findings* (2nd ed.). Thousand Oaks, CA: Sage.

McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. *Journal of Applied Psychology, 79*(4), 599–616.

Schmidt, F. L., & Hunter, J. E. (2014). *Methods of meta-analysis: Correcting error and bias in research findings* (3rd ed.). Thousand Oaks, CA: Sage.

tips/hunter_schmidt_method.txt · Last modified: 2019/01/17 12:26 (external edit)

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International