In chest X-ray (CXR) image analysis, rule-based systems are usually employed to extract labels from reports for dataset releases. However, there is still room for improvement in label quality. These labelers typically output only presence labels, sometimes with binary uncertainty indicators, which limits their usefulness. Supervised deep learning models have also been developed for report labeling but lack adaptability, similar to rule-based systems. In this work, we present MAPLEZ (Medical report Annotations with Privacy-preserving Large language model using Expeditious Zero shot answers), a novel approach leveraging a locally executable Large Language Model (LLM) to extract and enhance findings labels on CXR reports. MAPLEZ extracts not only binary labels indicating the presence or absence of a finding but also the location, severity, and radiologists’ uncertainty about the finding. Over eight abnormalities from five test sets, we show that our method can extract these annotations with an increase of 3.6 percentage points (pp) in macro F1 score for categorical presence annotations and more than 20 pp increase in F1 score for the location annotations over competing labelers. Additionally, using the combination of improved annotations and multi-type annotations in classification supervision in a dataset of limited-resolution CXRs, we demonstrate substantial advancements in proof-of-concept classification quality, with an increase of 1.1 pp in AUROC over models trained with annotations from the best alternative approach. We share code and annotations.

中文翻译：

---

使用保护隐私的大型语言模型和多类型注释增强胸部 X 光数据集：一种改进分类的数据驱动方法


在胸部 X 光 （CXR） 图像分析中，通常采用基于规则的系统从报告中提取标签以进行数据集发布。然而，标签质量仍有改进的空间。这些贴标机通常只输出存在标签，有时还带有二进制不确定性指示符，这限制了它们的实用性。监督式深度学习模型也被开发用于报告标记，但缺乏适应性，类似于基于规则的系统。在这项工作中，我们提出了 MAPLEZ（使用快速零镜头答案的具有隐私保护大语言模型的医学报告注释），这是一种利用本地可执行的大语言模型 （LLM） 来提取和增强 CXR 报告上的发现标签的新方法。MAPLEZ 不仅提取指示存在与否发现的二元标签，还提取位置、严重性和放射科医生对发现的不确定性。在来自五个测试集的 8 个异常中，我们表明我们的方法可以提取这些注释，与竞争标签商相比，分类存在注释的宏 F1 分数增加了 3.6 个百分点 （pp），位置注释的 F1 分数增加了 20 pp 以上。此外，在有限分辨率 CXR 数据集中，在分类监督中结合使用改进的注释和多类型注释，我们展示了概念验证分类质量的重大进步，与使用最佳替代方法的注释训练的模型相比，AUROC 增加了 1.1 pp。我们共享代码和注释。