3D street scene semantic segmentation is essential for urban understanding. However, supervised point cloud semantic segmentation networks heavily rely on expensive manual annotations and demonstrate limited generalization capabilities across datasets, which poses limitations in a range of downstream tasks. In contrast, image segmentation networks exhibit stronger generalization. Fortunately, mobile laser scanning systems can collect images and point clouds simultaneously, offering a potential solution for 2D-3D semantic transfer. In this paper, we introduce a cross-modal label transfer framework for point cloud semantic segmentation, without the supervision of 3D semantic annotation. Specifically, the proposed method takes point clouds and the associated posed images of a scene as inputs, and accomplishes the pointwise semantic segmentation for point clouds. We first get the image semantic pseudo-labels through a pre-trained image semantic segmentation model. Building on this, we construct implicit neural radiance fields (NeRF) to achieve multi-view consistent label mapping by jointly constructing color and semantic fields. Then, we design a superpoint semantic module to capture the local geometric features on point clouds, which contributes a lot to correcting semantic errors in the implicit field. Moreover, we introduce a dynamic object filter and a pose adjustment module to address the spatio-temporal misalignment between point clouds and images, further enhancing the consistency of the transferred semantic labels. The proposed approach has shown promising outcomes on two street scene datasets, namely KITTI-360 and WHU-Urban3D, highlighting the effectiveness and reliability of our method. Compared to the SoTA point cloud semantic segmentation method, namely SPT, the proposed method improves mIoU by approximately 15% on the WHU-Urban3D dataset. Our code and data are available at https://github.com/a4152684/StreetSeg.

中文翻译：

---

用于点云语义分割的跨模态语义迁移


3D 街景语义分割对于城市理解至关重要。然而，监督点云语义分割网络严重依赖昂贵的手动注释，并且跨数据集的泛化能力有限，这在一系列下游任务中构成了限制。相比之下，图像分割网络表现出更强的泛化。幸运的是，移动激光扫描系统可以同时收集图像和点云，为 2D-3D 语义迁移提供了潜在的解决方案。在本文中，我们介绍了一种用于点云语义分割的跨模态标签转移框架，无需 3D 语义注释的监督。具体来说，所提出的方法将点云和场景的相关姿势图像作为输入，并完成点云的逐点语义分割。我们首先通过预先训练的图像语义分割模型来获取图像语义伪标签。在此基础上，我们构建了隐式神经辐射场 （NeRF），通过联合构建颜色场和语义场来实现多视图一致的标签映射。然后，我们设计了一个超点语义模块来捕获点云上的局部几何特征，这对纠正隐含域中的语义错误有很大帮助。此外，我们引入了动态对象过滤器和姿态调整模块来解决点云和图像之间的时空错位问题，进一步增强了传递语义标签的一致性。所提出的方法在两个街景数据集上显示出有希望的结果，即 KITTI-360 和 WHU-Urban3D，突出了我们方法的有效性和可靠性。 与 SoTA 点云语义分割方法（即 SPT）相比，所提出的方法在 WHU-Urban3D 数据集上将 mIoU 提高了约 15%。我们的代码和数据可在 https://github.com/a4152684/StreetSeg 上获得。