Class-Aware Generative Adversarial Transformers for Medical Image Segmentation

by   Chenyu You, et al.

Transformers have made remarkable progress towards modeling long-range dependencies within the medical image analysis domain. However, current transformer-based models suffer from several disadvantages: (1) existing methods fail to capture the important features of the images due to the naive tokenization scheme; (2) the models suffer from information loss because they only consider single-scale feature representations; and (3) the segmentation label maps generated by the models are not accurate enough without considering rich semantic contexts and anatomical textures. In this work, we present CA-GANformer, a novel type of generative adversarial transformers, for medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design a novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures. Lastly, we utilize an adversarial training strategy that boosts segmentation accuracy and correspondingly allows a transformer-based discriminator to capture high-level semantically correlated contents and low-level anatomical features. Our experiments demonstrate that CA-GANformer dramatically outperforms previous state-of-the-art transformer-based approaches on three benchmarks, obtaining 2.54 qualitative experiments provide a more detailed picture of the model's inner workings, shed light on the challenges in improved transparency, and demonstrate that transfer learning can greatly improve performance and reduce the size of medical image datasets in training, making CA-GANformer a strong starting point for downstream medical image analysis tasks. Codes and models will be available to the public.


page 2

page 6

page 7

page 12

page 16

page 17


HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentation

Convolutional neural networks (CNNs) have been the consensus for medical...

Pyramid Medical Transformer for Medical Image Segmentation

Deep neural networks have been a prevailing technique in the field of me...

Implicit Anatomical Rendering for Medical Image Segmentation with Stochastic Experts

Integrating high-level semantically correlated contents and low-level an...

Fabric Image Representation Encoding Networks for Large-scale 3D Medical Image Analysis

Deep neural networks are parameterised by weights that encode feature re...

ScaleFormer: Revisiting the Transformer-based Backbones from a Scale-wise Perspective for Medical Image Segmentation

Recently, a variety of vision transformers have been developed as their ...

Generative Adversarial Transformers

We introduce the GANsformer, a novel and efficient type of transformer, ...

AerialFormer: Multi-resolution Transformer for Aerial Image Segmentation

Aerial Image Segmentation is a top-down perspective semantic segmentatio...

Please sign up or login with your details

Forgot password? Click here to reset