BT-GANformer: A generative ensemble transformer mechanism for brain tumor segmentation and classification

Abstract

The segmentation task for brain tumors from Magnetic Resonance Imaging (MRI) has been challenging and crucial to radiologists in their decision-making process. The recent developments in the attention mechanism in Natural Language Processing tasks have gained wide popularity and potential applications in Computer Vision and related problems. This article proposes a Generative ensembled Vision Transformer that achieves a State-of-the-Art (SOTA) performance in segmenting Brain tumors from multiple modalities of MRI scans. The proposed method includes an encoder and decoder block with CNN and Transformer, which forms the Generative architecture. The discriminator distinguishes the predictions of the Generator from the ground truth and consists of convolution layers along with a softmax for the classification tasks. The model was trained using the BraTS 2021 Task 1 dataset for the segmentation, and the Task 2 dataset was applied to evaluate the classification task. The proposed model scores a DICE average of 91% with a phenomenal score in tumor-core (TC), enhancing-tumor (ET), and whole-tumor (WT) categories. The model scores 99% ROC AUC score in the methylguanine‐methyltransferase (MGMT) classification task.