Reconstructing semantic indoor scenes is a challenging task in augmented and virtual reality. The quality of scene reconstruction is limited by the complexity and occlusion of indoor scenes. This is due to the difficulty in estimating the spatial structure of the scene and insufficient learning for object location inference. To address these challenges, we have developed PesRec, an end-to-end multi-task scene reconstruction network for parameterizing indoor semantic information. PesRec incorporates a newly designed spatial layout estimator and a 3D object detector to effectively learn scene parameter features from RGB images. We modify an object mesh generator to enhance the robustness of reconstructing indoor occluded objects through point cloud optimization in PesRec. Using the analyzed scene parameters and spatial structure, the proposed PesRec reconstructs an indoor scene by placing object meshes scaled to 3D detection boxes in an estimated layout cuboid. Extensive experiments on two benchmark datasets demonstrate that PesRec performs exceptionally well for object reconstruction with an average chamfer distance of 5.24 × 10-3 on the Pix3D dataset including 53.61 % mAP for 3D object detection and 79.7 % 3D IoU for the estimation of layout on the commonly-used SUN RGB-D datasets. The proposed computing network breaks through the limitations caused by complex indoor scenes and occlusions, showing optimization results that improve the quality of reconstruction in the fields of augmented reality and virtual reality.

中文翻译：

---

PesRec：一种从单幅图像重建室内语义场景的参数估计方法


在增强现实和虚拟现实中重建语义室内场景是一项具有挑战性的任务。场景重建的质量受到室内场景的复杂性和遮挡的限制。这是由于场景的空间结构难以估计以及对象位置推断的学习不足造成的。为了应对这些挑战，我们开发了 PesRec，一种用于参数化室内语义信息的端到端多任务场景重建网络。 PesRec 结合了新设计的空间布局估计器和 3D 对象检测器，可以有效地从 RGB 图像中学习场景参数特征。我们修改了对象网格生成器，以通过 PesRec 中的点云优化来增强重建室内遮挡对象的鲁棒性。使用分析的场景参数和空间结构，所提出的 PesRec 通过将缩放到 3D 检测框的对象网格放置在估计的布局长方体中来重建室内场景。在两个基准数据集上进行的大量实验表明，PesRec 在对象重建方面表现出色，在 Pix3D 数据集上的平均倒角距离为 5.24 × 10-3，其中用于 3D 对象检测的 mAP 为 53.61%，用于估计布局的 3D IoU 为 79.7%。常用的 SUN RGB-D 数据集。所提出的计算网络突破了复杂室内场景和遮挡造成的限制，显示出提高增强现实和虚拟现实领域重建质量的优化结果。