This article illustrates novel quantitative methods to estimate classification consistency in machine learning models used for screening measures. Screening measures are used in psychology and medicine to classify individuals into diagnostic classifications. In addition to achieving high accuracy, it is ideal for the screening process to have high classification consistency, which means that respondents would be classified into the same group every time if the assessment was repeated. Although machine learning models are increasingly being used to predict a screening classification based on individual item responses, methods to describe the classification consistency of machine learning models have not yet been developed. This article addresses this gap by describing methods to estimate classification inconsistency in machine learning models arising from two different sources: sampling error during model fitting and measurement error in the item responses. These methods use data resampling techniques such as the bootstrap and Monte Carlo sampling. These methods are illustrated using three empirical examples predicting a health condition/diagnosis from item responses. R code is provided to facilitate the implementation of the methods. This article highlights the importance of considering classification consistency alongside accuracy when studying screening measures and provides the tools and guidance necessary for applied researchers to obtain classification consistency indices in their machine learning research on diagnostic assessments. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

中文翻译：

---

估计机器学习模型的分类一致性以进行筛选措施。


本文阐述了估计用于筛选措施的机器学习模型中分类一致性的新颖定量方法。心理学和医学中使用筛选措施将个体分类为诊断类别。除了达到较高的准确性外，筛选过程最好具有较高的分类一致性，这意味着如果重复评估，每次都会将受访者分为同一组。尽管机器学习模型越来越多地用于根据单个项目响应来预测筛选分类，但描述机器学习模型分类一致性的方法尚未开发出来。本文通过描述估计机器学习模型中由两个不同来源引起的分类不一致的方法来解决这一差距：模型拟合期间的采样误差和项目响应中的测量误差。这些方法使用数据重采样技术，例如自举和蒙特卡罗采样。这些方法使用三个根据项目响应预测健康状况/诊断的经验示例进行说明。提供R代码以方便方法的实现。本文强调了在研究筛选措施时考虑分类一致性和准确性的重要性，并为应用研究人员在诊断评估的机器学习研究中获得分类一致性指数提供了必要的工具和指导。 （PsycInfo 数据库记录 (c) 2024 APA，保留所有权利）。