Automated radiology report generation has the potential to improve patient care and reduce the workload of radiologists. However, the path toward real-world adoption has been stymied by the challenge of evaluating the clinical quality of artificial intelligence (AI)-generated reports. We build a state-of-the-art report generation system for chest radiographs, called Flamingo-CXR, and perform an expert evaluation of AI-generated reports by engaging a panel of board-certified radiologists. We observe a wide distribution of preferences across the panel and across clinical settings, with 56.1% of Flamingo-CXR intensive care reports evaluated to be preferable or equivalent to clinician reports, by half or more of the panel, rising to 77.7% for in/outpatient X-rays overall and to 94% for the subset of cases with no pertinent abnormal findings. Errors were observed in human-written reports and Flamingo-CXR reports, with 24.8% of in/outpatient cases containing clinically significant errors in both report types, 22.8% in Flamingo-CXR reports only and 14.0% in human reports only. For reports that contain errors we develop an assistive setting, a demonstration of clinician–AI collaboration for radiology report composition, indicating new possibilities for potential clinical utility.

自动生成放射学报告有可能改善患者护理并减少放射科医生的工作量。然而，由于评估人工智能 （AI） 生成的报告的临床质量的挑战，通往现实世界采用的道路受到了阻碍。我们建立了一个最先进的胸片报告生成系统，称为 Flamingo-CXR，并通过聘请一个由委员会认证的放射科医生组成的小组对 AI 生成的报告进行专家评估。我们观察到整个面板和临床环境的偏好分布很广，56.1% 的 Flamingo-CXR 重症监护报告被评估为优于或等效于临床医生报告，面板的一半或更多，总体上/门诊 X 光检查上升到 77.7%，没有相关异常发现的病例子集上升到 94%。在人工编写的报告和 Flamingo-CXR 报告中观察到错误，24.8% 的住院/门诊病例在两种报告类型中都包含临床显着错误，22.8% 仅在 Flamingo-CXR 报告中，14.0% 仅在人工报告中。对于包含错误的报告，我们开发了一个辅助设置，展示了临床医生与 AI 协作撰写放射学报告，表明了潜在临床用途的新可能性。