Multimodal machine learning models are being developed to analyze pathology images and other modalities, such as gene expression, to gain clinical and biological insights. However, most frameworks for multimodal data fusion do not fully account for the interactions between different modalities. Here, we present an attention-based fusion architecture that integrates a graph representation of pathology images with gene expression data and concomitantly learns from the fused information to predict patient-specific survival. In our approach, pathology images are represented as undirected graphs, and their embeddings are combined with embeddings of gene expression signatures using an attention mechanism to stratify tumors by patient survival. We show that our framework improves the survival prediction of human non-small cell lung cancers, outperforming existing state-of-the-art approaches that leverage multimodal data. Our framework can facilitate spatial molecular profiling to identify tumor heterogeneity using pathology images and gene expression data, complementing results obtained from more expensive spatial transcriptomic and proteomic technologies.

中文翻译：

---

基于注意力的病理图像和基因表达融合图，用于预测癌症生存率


正在开发多模态机器学习模型来分析病理图像和其他模态（例如基因表达），以获得临床和生物学见解。然而，大多数多模态数据融合框架并没有完全考虑不同模态之间的交互。在这里，我们提出了一种基于注意力的融合架构，它将病理图像的图形表示与基因表达数据相结合，并同时从融合的信息中学习以预测患者特异性的生存率。在我们的方法中，病理图像表示为无向图，它们的嵌入与基因表达特征的嵌入相结合，使用注意力机制按患者生存率对肿瘤进行分层。我们表明，我们的框架改进了人类非小细胞肺癌的生存预测，优于利用多模态数据的现有最先进的方法。我们的框架可以促进空间分子分析，以使用病理图像和基因表达数据识别肿瘤异质性，补充从更昂贵的空间转录组学和蛋白质组学技术获得的结果。