Design of an iterative method for skin disease classification integrating multimodal data fusion with MVAE and transfer learning via Inception-ResNet V2

Abstract

 This study deals with the necessity of advancement in the classification of skin diseases, attaining a classification result with optimized parameters, using soft computing, machine learning (ML), deep learning (DL), data science, and data analysis techniques. Conventional approaches require considerable amounts of labelled data, which are resource consuming when compared to other medical fields. To address these challenges, our work, by adopting an integrative methodology, introduces an integrative framework by utilizing multimodal data fusion, transfer learning with pre-trained models, uncertainty quantification, and active learning strategies. Our multimodal data fusion approach is based on the multimodal variational autoencoder (MVAE), a powerful method for obtaining joint latent representations from diverse data modalities, including images, textual descriptions, patient histories, and genetic information. This method highly outperforms the single-modality approaches, especially in improving classification accuracy metrics such as F1-scores and area under the ROC curve (AUC). In addition, we make use of fine-tuning the pre-trained Inception-ResNet V2 model for transfer learning as a way of enhancing the capacity to classify skin diseases. Our methodology introduces the Monte Carlo dropout Bayesian convolutional neural network (MC-Bayes CNN) for uncertainty quantification. This novel approach, for the first time, allows us to make predictions with probabilistic values, including uncertainties, an extremely important development for the application of medicine to the diagnosis of diseases. Finally, the incorporation of collaboration-by-committee (QBC) active learning with Bayesian neural networks is expected to significantly revolutionize efficient model training with minimal labelled data samples. This indeed reduces the amount of labelled data needed; thereby significantly enhancing the classification accuracy achieved with only limited labelled data samples.