We present a novel neural algorithm for performing high-quality, highresolution, real-time novel view synthesis. From a sparse set of input RGB images or videos streams, our network both reconstructs the 3D scene and renders novel views at 1080p resolution at 30fps on an NVIDIA A100. Our feed-forward network generalizes across a wide variety of datasets and scenes and produces state-of-the-art quality for a real-time method. Our quality approaches, and in some cases surpasses, the quality of some of the top offline methods. In order to achieve these results we use a novel combination of several key concepts, and tie them together into a cohesive and effective algorithm. We build on previous works that represent the scene using semi-transparent layers and use an iterative learned render-and-refine approach to improve those layers. Instead of flat layers, our method reconstructs layered depth maps (LDMs) that efficiently represent scenes with complex depth and occlusions. The iterative update steps are embedded in a multi-scale, UNet-style architecture to perform as much compute as possible at reduced resolution. Within each update step, to better aggregate the information from multiple input views, we use a specialized Transformer-based network component. This allows the majority of the per-input image processing to be performed in the input image space, as opposed to layer space, further increasing efficiency. Finally, due to the real-time nature of our reconstruction and rendering, we dynamically create and discard the internal 3D geometry for each frame, generating the LDM for each view. Taken together, this produces a novel and effective algorithm for view synthesis. Through extensive evaluation, we demonstrate that we achieve state-of-the-art quality at real-time rates.

中文翻译：

---

Quark：实时、高分辨率和通用神经视图合成


我们提出了一种新的神经算法，用于执行高质量、高分辨率、实时的新视图合成。我们的网络从一组稀疏的输入 RGB 图像或视频流中，在 NVIDIA A100 上以 1080p 分辨率和 30fps 渲染新颖的视图。我们的前馈网络泛化到各种数据集和场景中，并为实时方法提供最先进的质量。我们的质量接近，在某些情况下甚至超过了一些顶级离线方法的质量。为了实现这些结果，我们使用了几个关键概念的新颖组合，并将它们结合在一起形成一个有凝聚力且有效的算法。我们以以前的作品为基础，使用半透明图层来表现场景，并使用迭代学习的渲染和优化方法来改进这些图层。我们的方法不是平面层，而是重建分层深度图 （LDM），可以有效地表示具有复杂深度和遮挡的场景。迭代更新步骤嵌入在多尺度 UNet 样式架构中，以降低的分辨率执行尽可能多的计算。在每个更新步骤中，为了更好地聚合来自多个 Importing 视图的信息，我们使用了一个专门的基于 Transformer 的网络组件。这允许在输入图像空间（而不是图层空间）中执行大部分每个输入的图像处理，从而进一步提高效率。最后，由于重建和渲染的实时性，我们动态地为每个帧创建和丢弃内部 3D 几何体，为每个视图生成 LDM。总之，这产生了一种新颖而有效的视图合成算法。 通过广泛的评估，我们证明了我们以实时速率实现了最先进的质量。