The last few years have seen a boom in using generative models to augment real datasets, as synthetic data can effectively model real data distributions and provide privacy-preserving, shareable datasets that can be used to train deep learning models. However, most of these methods are 2D and provide synthetic datasets that come, at most, with categorical annotations. The generation of paired images and segmentation samples that can be used in downstream, supervised segmentation tasks remains fairly uncharted territory. This work proposes a two-stage generative model capable of producing 2D and 3D semantic label maps and corresponding multi-modal images. We use a latent diffusion model for label synthesis and a VAE-GAN for semantic image synthesis. Synthetic datasets provided by this model are shown to work in a wide variety of segmentation tasks, supporting small, real datasets or fully replacing them while maintaining good performance. We also demonstrate its ability to improve downstream performance on out-of-distribution data.

中文翻译：

---

为脑部 MRI 生成多病理和多模态图像和标签


过去几年，使用生成模型来增强真实数据集的繁荣，因为合成数据可以有效地对真实数据分布进行建模，并提供可用于训练深度学习模型的保护隐私、可共享的数据集。但是，这些方法中的大多数都是 2D 的，并且提供最多带有分类注释的合成数据集。生成可用于下游监督分割任务的配对图像和分割样本仍然是一个相当未知的领域。这项工作提出了一种两阶段生成模型，能够生成 2D 和 3D 语义标签映射以及相应的多模态图像。我们使用潜在扩散模型进行标签合成，使用 VAE-GAN 进行语义图像合成。此模型提供的合成数据集被证明可以在各种分割任务中工作，支持小型真实数据集或完全替换它们，同时保持良好的性能。我们还展示了它提高 out-of-distribution 数据的下游性能的能力。