Vision Transformers recently achieved a competitive performance compared with CNNs due to their excellent capability of learning global representation. However, there are two major challenges when applying them to 3D image segmentation: i) Because of the large size of 3D medical images, comprehensive global information is hard to capture due to the enormous computational costs. ii) Insufficient local inductive bias in Transformers affects the ability to segment detailed features such as ambiguous and subtly defined boundaries. Hence, to apply the Vision Transformer mechanism in the medical image segmentation field, the above challenges need to be overcome adequately. We propose a hybrid paradigm, called Variable-Shape Mixed Transformer (VSmTrans), that integrates self-attention and convolution and can enjoy the benefits of free learning of both complex relationships from the self-attention mechanism and the local prior knowledge from convolution. Specifically, we designed a Variable-Shape self-attention mechanism, which can rapidly expand the receptive field without extra computing cost and achieve a good trade-off between global awareness and local details. In addition, the parallel convolution paradigm introduces strong local inductive bias to facilitate the ability to excavate details. Meanwhile, a pair of learnable parameters can automatically adjust the importance of the above two paradigms. Extensive experiments were conducted on two public medical image datasets with different modalities: the AMOS CT dataset and the BraTS2021 MRI dataset. Our method achieves the best average Dice scores of 88.3 % and 89.7 % on these datasets, which are superior to the previous state-of-the-art Swin Transformer-based and CNN-based architectures. A series of ablation experiments were also conducted to verify the efficiency of the proposed hybrid mechanism and the components and explore the effectiveness of those key parameters in VSmTrans. The proposed hybrid Transformer-based backbone network for 3D medical image segmentation can tightly integrate self-attention and convolution to exploit the advantages of these two paradigms. The experimental results demonstrate our method's superiority compared to other state-of-the-art methods. The hybrid paradigm seems to be most appropriate to the medical image segmentation field. The ablation experiments also demonstrate that the proposed hybrid mechanism can effectively balance large receptive fields with local inductive biases, resulting in highly accurate segmentation results, especially in capturing details. Our code is available at https://github.com/qingze-bai/VSmTrans.

中文翻译：

---

VSmTrans：一种集成自注意力和卷积的混合范式，用于 3D 医学图像分割


与 CNN 相比，Vision Transformers 最近取得了具有竞争力的性能，因为他们具有出色的学习全球代表的能力。然而，将它们应用于 3D 图像分割时存在两个主要挑战：i） 由于 3D 医学图像的尺寸很大，由于计算成本巨大，难以捕获全面的全球信息。ii） 变压器中局部电感偏置不足会影响分割详细特征的能力，例如模棱两可和定义微妙的边界。因此，要将 Vision Transformer 机制应用于医学图像分割领域，需要充分克服上述挑战。我们提出了一种混合范式，称为 Variable-Shape Mixed Transformer （VSmTrans），它集成了自我注意和卷积，并且可以享受从自我注意机制中自由学习复杂关系和从卷积中自由学习局部先验知识的好处。具体来说，我们设计了一个 Variable-Shape self-attention 机制，它可以在没有额外计算成本的情况下快速扩展感受野，并在全局感知和局部细节之间实现良好的权衡。此外，并行卷积范式引入了强大的局部归纳偏差，以促进挖掘细节的能力。同时，一对可学习的参数可以自动调整上述两种范式的重要性。对两个不同模式的公共医学图像数据集进行了广泛的实验： AMOS CT 数据集和 BraTS2021 MRI 数据集。我们的方法在这些数据集上实现了 88.3% 和 89.7% 的最佳平均 Dice 分数，这优于以前最先进的基于 Swin Transformer 和基于 CNN 的架构。 还进行了一系列消融实验，以验证所提出的混合机制和组件的效率，并探索这些关键参数在 VSmTrans 中的有效性。所提出的用于 3D 医学图像分割的基于 Transformer 的混合骨干网络可以紧密集成自注意力和卷积，以利用这两种范式的优势。实验结果表明，与其他最先进的方法相比，我们的方法具有优越性。混合范式似乎最适合医学图像分割领域。消融实验还表明，所提出的混合机制可以有效地平衡大感受野与局部感应偏置，从而获得高度准确的分割结果，尤其是在捕获细节方面。我们的代码可在 https://github.com/qingze-bai/VSmTrans 获取。