Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a Colonoscopy 3D Video Dataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at https://durr.jhu.edu/C3VD.

筛查结肠镜检查是多种 3D 计算机视觉技术的重要临床应用，包括深度估计、表面重建和缺失区域检测。然而，由于难以获取真实数据，这些技术在真实结肠镜检查视频中的开发、评估和比较在很大程度上仍然是定性的。在这项工作中，我们提出了通过高清临床结肠镜和高保真结肠模型获取的结肠镜检查3D视频数据集 (C3VD)，用于对结肠镜检查中的计算机视觉方法进行基准测试。我们引入了一种新颖的多模态 2D-3D 配准技术，将光学视频序列与已知 3D 模型的真实渲染视图配准。通过使用生成对抗网络将光学图像转换为深度图并使用进化优化器对齐边缘特征来注册不同的模态。在无误差地面实况的模拟实验中，该配准方法实现了 0.321 毫米的平均平移误差和 0.159 度的平均旋转误差。该方法还利用视频信息，与单帧配准相比，平移配准精度提高了 55.6%，旋转配准精度提高了 60.4%。注册了 22 个短视频序列，生成总共 10,015 个帧，其中包含配对的地面真实深度、表面法线、光流、遮挡、六自由度姿势、覆盖图和 3D 模型。该数据集还包括胃肠病学家获取的筛选视频以及配对的地面真实姿势和 3D 表面模型。数据集和注册源代码可在https://durr.jhu.edu/C3VD获取。