Multi-view projection techniques have shown themselves to be highly effective in achieving top-performing results in the recognition of 3D shapes. These methods involve learning how to combine information from multiple view-points. However, the camera view-points from which these views are obtained are often fixed for all shapes. To overcome the static nature of current multi-view techniques, we propose learning these view-points. Specifically, we introduce the Multi-View Transformation Network (MVTN), which uses differentiable rendering to determine optimal view-points for 3D shape recognition. As a result, MVTN can be trained end-to-end with any multi-view network for 3D shape classification. We integrate MVTN into a novel adaptive multi-view pipeline that is capable of rendering both 3D meshes and point clouds. Our approach demonstrates state-of-the-art performance in 3D classification and shape retrieval on several benchmarks (ModelNet40, ScanObjectNN, ShapeNet Core55). Further analysis indicates that our approach exhibits improved robustness to occlusion compared to other methods. We also investigate additional aspects of MVTN, such as 2D pretraining and its use for segmentation. To support further research in this area, we have released MVTorch, a PyTorch library for 3D understanding and generation using multi-view projections.

多视图投影技术已证明，在识别 3D 形状方面非常有效。这些方法涉及学习如何组合来自多个视点的信息。但是，对于所有形状，从中获取这些视图的照相机视点通常是固定的。为了克服当前多视图技术的静态性质，我们建议学习这些视点。具体来说，我们介绍了多视图转换网络 （MVTN），它使用可微渲染来确定 3D 形状识别的最佳视点。因此，MVTN 可以使用任何多视图网络进行端到端训练，以进行 3D 形状分类。我们将 MVTN 集成到一个新颖的自适应多视图管道中，该管道能够渲染 3D 网格和点云。我们的方法在多个基准（ModelNet40、ScanObjectNN、ShapeNet Core55）上展示了 3D 分类和形状检索方面的最新性能。进一步的分析表明，与其他方法相比，我们的方法表现出更好的抗咬合鲁棒性。我们还研究了 MVTN 的其他方面，例如 2D 预训练及其在分割中的应用。为了支持该领域的进一步研究，我们发布了 MVTorch，这是一个 PyTorch 库，用于使用多视图投影进行 3D 理解和生成。