We propose a novel framework for detecting 3D human–object interactions (HOI) in construction sites and a toolkit for generating construction-related human–object interaction graphs. Computer vision methods have been adopted for construction site safety surveillance in recent years. The current computer vision methods rely on videos and images, with which safety verification is performed on common-sense knowledge, without considering 3D spatial relationships among the detected instances. We propose a new method to incorporate spatial understanding by directly inferring the interactions from 3D point cloud data. The proposed model is trained on a 3D construction site dataset generated from our crafted simulation toolkit. The model achieves 54.11% mean interaction over union (mIOU) and 72.98% average mean precision(mAP) for the worker–object interaction relationship recognition. The model is also validated on PiGraphs, a benchmarking dataset with 3D human–object interaction types, and compared against other existing 3D interaction detection frameworks. It was observed that it achieves superior performance from the state-of-the-art model, increasing the interaction detection mAP by 17.01%. Besides the 3D interaction model, we also simulate interactions from industrial surveillance footage using MoCap and physical constraints, which will be released to foster future studies in the domain.

中文翻译：

---

从可转移的上下文知识中学习 3D 人机交互图，用于施工监控


我们提出了一种用于检测建筑工地中 3D 人机交互 （HOI） 的新框架，以及用于生成与建筑相关的人机交互图的工具包。近年来，计算机视觉方法已被用于建筑工地安全监测。当前的计算机视觉方法依赖于视频和图像，这些视频和图像是根据常识性知识进行安全验证的，而不考虑检测到的实例之间的 3D 空间关系。我们提出了一种新方法，通过直接从 3D 点云数据推断交互来整合空间理解。所提出的模型是在我们精心制作的仿真工具包生成的 3D 建筑工地数据集上进行训练的。该模型在工人与对象交互关系识别方面实现了 54.11% 的平均工会交互 （mIOU） 和 72.98% 的平均精度 （mAP）。该模型还在 PiGraphs（具有 3D 人机交互类型的基准数据集）上进行了验证，并与其他现有的 3D 交互检测框架进行了比较。据观察，它从最先进的模型中获得了卓越的性能，将交互检测 mAP 提高了 17.01%。除了 3D 交互模型外，我们还使用 MoCap 和物理约束模拟了来自工业监控录像的交互，这些交互将发布以促进该领域的未来研究。