Given the centrality of the job performance construct to organizational researchers, it is critical to understand the reliability of the most common way it is operationalized in the literature. To this end, we conducted an updated meta-analysis on the interrater reliability of supervisory ratings of job performance (k = 132 independent samples) using a new meta-analytic procedure (i.e., the Morris estimator), which includes both within- and between-study variance in the calculation of study weights. An important benefit of this approach is that it prevents large-sample studies from dominating the results. In this investigation, we also examined different factors that may affect interrater reliability, including job complexity, managerial level, rating purpose, performance measure, and rater perspective. We found a higher interrater reliability estimate (r = .65) compared to previous meta-analyses on the topic, and our results converged with an important, but often neglected, finding from a previous meta-analysis by Conway and Huffcutt (1997), such that interrater reliability varies meaningfully by job type (r = .57 for managerial positions vs. r = .68 for nonmanagerial positions). Given this finding, we advise against the use of an overall grand mean of interrater reliability. Instead, we recommend using job-specific or local reliabilities for making corrections for attenuation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

中文翻译：

---

监督绩效评级的参与者间可靠性的最新元分析。


鉴于工作绩效结构对组织研究人员的中心地位，了解文献中最常见的实施方式的可靠性至关重要。为此，我们使用新的荟萃分析程序（即莫里斯估计量）对工作绩效监督评级的人际可靠性（k = 132个独立样本）进行了更新的荟萃分析，其中包括内部和之间-研究权重计算中的研究方差。这种方法的一个重要好处是它可以防止大样本研究主导结果。在这项调查中，我们还研究了可能影响评估者间可靠性的不同因素，包括工作复杂性、管理水平、评级目的、绩效衡量标准和评估者视角。与之前关于该主题的荟萃分析相比，我们发现了更高的参与者间可靠性估计 (r = .65)，并且我们的结果与 Conway 和 Huffcutt (1997) 之前的荟萃分析中的一个重要但经常被忽视的发现相一致，因此，不同工作类型的评估者间可靠性存在显着差异（管理职位的 r = .57 与非管理职位的 r = .68）。鉴于这一发现，我们建议不要使用评估者间可靠性的总体平均值。相反，我们建议使用特定于工作或本地的可靠性来校正衰减。 （PsycInfo 数据库记录 (c) 2024 APA，保留所有权利）。