The adoption of machine learning (ML) and, more specifically, deep learning (DL) applications into all major areas of our lives is underway. The development of trustworthy AI is especially important in medicine due to the large implications for patients’ lives. While trustworthiness concerns various aspects including ethical, transparency and safety requirements, we focus on the importance of data quality (training/test) in DL. Since data quality dictates the behaviour of ML products, evaluating data quality will play a key part in the regulatory approval of medical ML products. We perform a systematic review following PRISMA guidelines using the databases Web of Science, PubMed and ACM Digital Library. We identify 5408 studies, out of which 120 records fulfil our eligibility criteria. From this literature, we synthesise the existing knowledge on data quality frameworks and combine it with the perspective of ML applications in medicine. As a result, we propose the METRIC-framework, a specialised data quality framework for medical training data comprising 15 awareness dimensions, along which developers of medical ML applications should investigate the content of a dataset. This knowledge helps to reduce biases as a major source of unfairness, increase robustness, facilitate interpretability and thus lays the foundation for trustworthy AI in medicine. The METRIC-framework may serve as a base for systematically assessing training datasets, establishing reference datasets, and designing test datasets which has the potential to accelerate the approval of medical ML products.

机器学习 (ML)，更具体地说，深度学习 (DL) 应用程序正在进入我们生活的所有主要领域。由于对患者生活的重大影响，值得信赖的人工智能的发展在医学领域尤其重要。虽然可信度涉及道德、透明度和安全要求等各个方面，但我们重点关注深度学习中数据质量（训练/测试）的重要性。由于数据质量决定了机器学习产品的行为，因此评估数据质量将在医疗机器学习产品的监管审批中发挥关键作用。我们遵循 PRISMA 指南，使用数据库 Web of Science、PubMed 和 ACM Digital Library 进行系统评价。我们确定了 5408 项研究，其中 120 项记录符合我们的资格标准。从这些文献中，我们综合了有关数据质量框架的现有知识，并将其与机器学习在医学中应用的角度相结合。因此，我们提出了 METRIC 框架，这是一个专门用于医疗培训数据的数据质量框架，包含 15 个认知维度，医疗 ML 应用程序的开发人员应沿着该框架调查数据集的内容。这些知识有助于减少作为不公平主要来源的偏见，提高鲁棒性，促进可解释性，从而为医学领域值得信赖的人工智能奠定基础。 METRIC框架可以作为系统评估训练数据集、建立参考数据集和设计测试数据集的基础，这有可能加速医疗机器学习产品的批准。