The last few years have seen a boom in using generative models to augment real datasets, as synthetic data can effectively model real data distributions and provide privacy-preserving, shareable datasets that can be used to train deep learning models. However, most of these methods are 2D and provide synthetic datasets that come, at most, with categorical annotations. The generation of paired images and segmentation samples that can be used in downstream, supervised segmentation tasks remains fairly uncharted territory. This work proposes a two-stage generative model capable of producing 2D and 3D semantic label maps and corresponding multi-modal images. We use a latent diffusion model for label synthesis and a VAE-GAN for semantic image synthesis. Synthetic datasets provided by this model are shown to work in a wide variety of segmentation tasks, supporting small, real datasets or fully replacing them while maintaining good performance. We also demonstrate its ability to improve downstream performance on out-of-distribution data.

中文翻译：

---

生成脑 MRI 的多病理和多模态图像和标签


过去几年出现了使用生成模型来增强真实数据集的热潮，因为合成数据可以有效地对真实数据分布进行建模，并提供可用于训练深度学习模型的保护隐私、可共享的数据集。然而，这些方法大多数都是二维的，并提供最多带有分类注释的合成数据集。可用于下游监督分割任务的配对图像和分割样本的生成仍然是相当未知的领域。这项工作提出了一种两阶段生成模型，能够生成 2D 和 3D 语义标签图以及相应的多模态图像。我们使用潜在扩散模型进行标签合成，使用 VAE-GAN 进行语义图像合成。该模型提供的合成数据集可用于各种分割任务，支持小型真实数据集或完全替换它们，同时保持良好的性能。我们还展示了它提高分布外数据下游性能的能力。