This position paper argues for the use of structured generative models (SGMs) for the understanding of static scenes. This requires the reconstruction of a 3D scene from an input image (or a set of multi-view images), whereby the contents of the image(s) are causally explained in terms of models of instantiated objects, each with their own type, shape, appearance and pose, along with global variables like scene lighting and camera parameters. This approach also requires scene models which account for the co-occurrences and inter-relationships of objects in a scene. The SGM approach has the merits that it is compositional and generative, which lead to interpretability and editability. To pursue the SGM agenda, we need models for objects and scenes, and approaches to carry out inference. We first review models for objects, which include “things” (object categories that have a well defined shape), and “stuff” (categories which have amorphous spatial extent). We then move on to review scene models which describe the inter-relationships of objects. Perhaps the most challenging problem for SGMs is inference of the objects, lighting and camera parameters, and scene inter-relationships from input consisting of a single or multiple images. We conclude with a discussion of issues that need addressing to advance the SGM agenda.

本立场文件主张使用结构化生成模型 （SGM） 来理解静态场景。这需要从输入图像（或一组多视图图像）重建 3D 场景，其中图像的内容根据实例化对象的模型进行因果解释，每个对象都有自己的类型、形状、外观和姿势，以及场景照明和相机参数等全局变量。此方法还需要场景模型，这些模型考虑了场景中对象的共现和相互关系。SGM 方法的优点是它是组合性和生成性的，这导致了可解释性和可编辑性。为了实现 SGM 议程，我们需要对象和场景的模型，以及执行推理的方法。我们首先回顾对象的模型，其中包括 “things” （具有明确定义形状的对象类别） 和 “stuff” （具有无定形空间范围的类别）。然后，我们继续回顾描述对象相互关系的场景模型。对于 SGM 来说，最具挑战性的问题可能是从由单个或多个图像组成的输入中推断对象、照明和相机参数以及场景相互关系。最后，我们讨论了推进 SGM 议程需要解决的问题。