结合超声和红外热图像对甲状腺结节进行分类的自适应多模态混合模型,BMC Bioinformatics

当前位置： X-MOL 学术 › BMC Bioinform. › 论文详情

Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)

结合超声和红外热图像对甲状腺结节进行分类的自适应多模态混合模型
BMC Bioinformatics ( IF 2.9 ) Pub Date : 2023-08-19 , DOI: 10.1186/s12859-023-05446-2
Na Zhang ₁ , Juan Liu ₁ , Yu Jin ₁ , Wensi Duan ₁ , Ziling Wu ₂ , Zhaohui Cai ₁ , Meng Wu ₂

Affiliation

超声波 (US) 和红外热成像 (IRT) 是医疗应用中广泛采用的两种非侵入性、无辐射且廉价的成像技术。超声成像获得的超声图像主要表达病灶的大小、形状、轮廓边界、回声等形态信息，而红外热成像获得的红外热图像主要描述其热力学功能信息。虽然区分良恶性甲状腺结节需要形态和功能信息，但目前的深度学习模型仅基于超声图像，使得一些形态变化不显着但功能变化显着的恶性结节可能被漏检。鉴于 US 和 IRT 图像通过不同的模态呈现甲状腺结节，我们提出了一种自适应多模态混合 (AmmH) 分类模型，该模型可以利用这两种图像类型的合并来实现卓越的分类性能。AmmH 方法涉及为每个模态数据构建一个混合单模态编码器模块，通过集成 CNN 模块和 Transformer 模块，促进局部和全局特征的提取。然后使用自适应模态权重生成网络对从两种模态提取的特征进行自适应加权，并使用自适应跨模态编码器模块进行融合。随后，融合的特征通过 MLP 对甲状腺结节进行分类。在收集的数据集上，我们的 AmmH 模型分别实现了 97.17% 和 97.38% 的 F1 和 F2 分数，显着优于单模态模型。四次消融实验的结果进一步证明了我们提出的方法的优越性。所提出的多模态模型从各种模态图像中提取特征，从而增强甲状腺结节描述的全面性。自适应模态权重生成网络能够自适应关注不同模态，通过自适应跨模态编码器促进使用自适应权重的特征融合。因此，该模型表现出了良好的分类性能，表明其作为一种无创、无辐射且经济有效的筛查工具的潜力，可用于区分良性和恶性甲状腺结节。源代码可在 https://github.com/wuliZN2020/AmmH 获取。

"点击查看英文标题和摘要"

An adaptive multi-modal hybrid model for classifying thyroid nodules by combining ultrasound and infrared thermal images

Two types of non-invasive, radiation-free, and inexpensive imaging technologies that are widely employed in medical applications are ultrasound (US) and infrared thermography (IRT). The ultrasound image obtained by ultrasound imaging primarily expresses the size, shape, contour boundary, echo, and other morphological information of the lesion, while the infrared thermal image obtained by infrared thermography imaging primarily describes its thermodynamic function information. Although distinguishing between benign and malignant thyroid nodules requires both morphological and functional information, present deep learning models are only based on US images, making it possible that some malignant nodules with insignificant morphological changes but significant functional changes will go undetected. Given the US and IRT images present thyroid nodules through distinct modalities, we proposed an Adaptive multi-modal Hybrid (AmmH) classification model that can leverage the amalgamation of these two image types to achieve superior classification performance. The AmmH approach involves the construction of a hybrid single-modal encoder module for each modal data, which facilitates the extraction of both local and global features by integrating a CNN module and a Transformer module. The extracted features from the two modalities are then weighted adaptively using an adaptive modality-weight generation network and fused using an adaptive cross-modal encoder module. The fused features are subsequently utilized for the classification of thyroid nodules through the use of MLP. On the collected dataset, our AmmH model respectively achieved 97.17% and 97.38% of F1 and F2 scores, which significantly outperformed the single-modal models. The results of four ablation experiments further show the superiority of our proposed method. The proposed multi-modal model extracts features from various modal images, thereby enhancing the comprehensiveness of thyroid nodules descriptions. The adaptive modality-weight generation network enables adaptive attention to different modalities, facilitating the fusion of features using adaptive weights through the adaptive cross-modal encoder. Consequently, the model has demonstrated promising classification performance, indicating its potential as a non-invasive, radiation-free, and cost-effective screening tool for distinguishing between benign and malignant thyroid nodules. The source code is available at https://github.com/wuliZN2020/AmmH .

更新日期：2023-08-20

点击分享查看原文

点击收藏

公开下载

阅读更多本刊新发论文本刊介绍/投稿指南