Semantic scene completion (SSC) aims to simultaneously perform scene completion (SC) and predict semantic categories of a 3D scene from a single depth and/or RGB image. Most existing SSC methods struggle to handle complex regions with multiple objects close to each other, especially for objects with reflective or dark surfaces. This primarily stems from two challenges: (1) the loss of geometric information due to the unreliability of depth values from sensors, and (2) the potential for semantic confusion when simultaneously predicting 3D shapes and semantic labels. To address these problems, we propose a Semantic-guided Semantic Scene Completion framework, dubbed SG-SSC, which involves Semantic-guided Fusion (SGF) and Volume-guided Semantic Predictor (VGSP). Guided by 2D semantic segmentation maps, SGF adaptively fuses RGB and depth features to compensate for the missing geometric information caused by the missing values in depth images, thus performing more robustly to unreliable depth information. VGSP exploits the mutual benefit between SC and SSC tasks, making SSC more focused on predicting the categories of voxels with high occupancy probabilities and also allowing SC to utilize semantic priors to better predict voxel occupancy. Experimental results show that SG-SSC outperforms existing state-of-the-art methods on the NYU, NYUCAD, and SemanticKITTI datasets. Models and code are available at https://github.com/aipixel/SG-SSC.

语义场景完成 （SSC） 旨在同时执行场景完成 （SC） 并从单个深度和/或 RGB 图像预测 3D 场景的语义类别。大多数现有的 SSC 方法都难以处理多个对象彼此靠近的复杂区域，尤其是对于具有反射或深色表面的物体。这主要源于两个挑战：（1） 由于传感器深度值的不可靠性而导致几何信息丢失，以及 （2） 在同时预测 3D 形状和语义标签时可能出现语义混淆。为了解决这些问题，我们提出了一个语义引导的语义场景完成框架，称为 SG-SSC，它涉及语义引导融合 （SGF） 和体积引导语义预测器 （VGSP）。在 2D 语义分割图的引导下，SGF 自适应地融合 RGB 和深度特征，以补偿深度图像中缺失值导致的几何信息缺失，从而对不可靠的深度信息执行更稳健。VGSP 利用了 SC 和 SSC 任务之间的互惠互利，使 SSC 更专注于预测具有高占用概率的体素类别，并允许 SC 利用语义先验来更好地预测体素占用率。实验结果表明，SG-SSC 在 NYU 、 NYUCAD 和 SemanticKITTI 数据集上优于现有的最先进方法。模型和代码可在 https://github.com/aipixel/SG-SSC 获取。