With the increasing number of remote sensing (RS) data sources, the joint utilization of multimodal data in Earth observation tasks has become a crucial research topic. As a typical representative of RS data, hyperspectral images (HSIs) provide accurate spectral information, while rich elevation information can be obtained from light detection and ranging (LiDAR) data. However, due to the significant differences in multimodal heterogeneous features, how to efficiently fuse HSI and LiDAR data remains one of the challenges faced by existing research. In addition, the edge contour information of images is not fully considered by existing methods, which can easily lead to performance bottlenecks. Thus, a joint classification network of HSI and LiDAR data based on Mamba (HLMamba) is proposed. Specifically, a gradient joint algorithm (GJA) is first performed on LiDAR data to obtain the edge contour data of the land distribution. Subsequently, a multimodal feature extraction module (MFEM) was proposed to capture the semantic features of HSI, LiDAR, and edge contour data. Then, to efficiently fuse multimodal features, a novel deep learning (DL) framework called Mamba, was introduced, and a multimodal Mamba fusion module (MMFM) was constructed. By efficiently modeling the long-distance dependencies of multimodal sequences, the MMFM can better explore the internal features of multimodal data and the interrelationships between modalities, thereby enhancing fusion performance. Finally, to validate the effectiveness of HLMamba, a series of experiments were conducted on three common HSI and LiDAR datasets. The results indicate that HLMamba has superior classification performance compared to other state-of-the-art DL methods. The source code of the proposed method will be available publicly at https://github.com/Dilingliao/HLMamba .

中文翻译：

---

基于Mamba的高光谱与激光雷达数据联合分类


随着遥感数据源数量的不断增加，多模态数据在对地观测任务中的联合利用已成为一个重要的研究课题。作为遥感数据的典型代表，高光谱图像（HSI）提供了准确的光谱信息，而光探测和测距（LiDAR）数据可以获取丰富的高程信息。然而，由于多模态异构特征的显着差异，如何有效融合HSI和LiDAR数据仍然是现有研究面临的挑战之一。此外，现有方法没有充分考虑图像的边缘轮廓信息，很容易导致性能瓶颈。因此，提出了一种基于Mamba（HLMamba）的HSI和LiDAR数据联合分类网络。具体来说，首先对LiDAR数据进行梯度联合算法（GJA）以获得土地分布的边缘轮廓数据。随后，提出了多模态特征提取模块（MFEM）来捕获HSI、LiDAR和边缘轮廓数据的语义特征。然后，为了有效融合多模态特征，引入了一种称为 Mamba 的新型深度学习（DL）框架，并构建了多模态 Mamba 融合模块（MMFM）。通过对多模态序列的长距离依赖性进行有效建模，MMFM可以更好地探索多模态数据的内部特征以及模态之间的相互关系，从而提高融合性能。最后，为了验证 HLMamba 的有效性，在三个常见的 HSI 和 LiDAR 数据集上进行了一系列实验。结果表明，与其他最先进的深度学习方法相比，HLMamba 具有卓越的分类性能。 该方法的源代码将在 https://github.com/Dilingliao/HLMamba 公开提供。