Deep-learning-enabled multimodal data fusion for lung disease classification

Abstract

The recent pandemic has revealed the urgent need for lung disease diagnosis at early stages in humans. Deep learning-based automatic diagnosis methods typically rely on single-modality data such as medical imaging. However, analysis of single modality data is not so reliable to diagnose the disease at its early stages. Clinical data, blood tests together with imaging methods are very powerful and reliable sources to detect the presence of disease in the human body. This study attempts to use medical imaging data with clinical information to develop a multimodal fusion approach to detect lung disease. Two architectures of multimodal network based on late and intermediate fusion is proposed. Besides, an approach of adapting batch size is also introduced. Experiments show that the performance of intermediate fusion is better than the late fusion model with both direct and adaptive batch size approach.

Keywords

• Multimodal fusion
• Lung disease
• Medical imaging
• Clinical data
• Deep learning