Machine learning models for the diagnosis of breast cancer can facilitate the prediction of cancer risk and subsequent patient management among other clinical tasks. For the models to impact clinical practice, they ought to follow standard workflows, help interpret mammography and ultrasound data, evaluate clinical contextual information, handle incomplete data and be validated in prospective settings. Here we report the development and testing of a multimodal model leveraging mammography and ultrasound modules for the stratification of breast cancer risk based on clinical metadata, mammography and trimodal ultrasound (19,360 images of 5,216 breasts) from 5,025 patients with surgically confirmed pathology across medical centres and scanner manufacturers. Compared with the performance of experienced radiologists, the model performed similarly at classifying tumours as benign or malignant and was superior at pathology-level differential diagnosis. With a prospectively collected dataset of 191 breasts from 187 patients, the overall accuracies of the multimodal model and of preliminary pathologist-level assessments of biopsied breast specimens were similar (90.1% vs 92.7%, respectively). Multimodal models may assist diagnosis in oncology.

用于诊断乳腺癌的机器学习模型可以促进癌症风险的预测和随后的患者管理以及其他临床任务。为了使模型影响临床实践，它们应该遵循标准工作流程，帮助解释乳房 X 光检查和超声数据，评估临床背景信息，处理不完整的数据，并在前瞻性环境中进行验证。在这里，我们报告了利用乳房 X 光检查和超声模块的多模态模型的开发和测试，该模型基于临床元数据、乳房 X 光检查和三峰超声（5,216 个乳房的 19,360 张图像）对乳腺癌风险进行分层来自医疗中心和扫描仪制造商的 5,025 名手术确诊病理的患者。与经验丰富的放射科医生的表现相比，该模型在将肿瘤分类为良性或恶性方面表现相似，并且在病理学水平鉴别诊断方面表现出色。通过前瞻性收集的 187 名患者的 191 个乳房数据集，多模态模型和活检乳腺标本的初步病理学家水平评估的总体准确性相似（分别为 90.1% 和 92.7%）。多模态模型可能有助于肿瘤学的诊断。