GIAE-Net: A gradient-intensity oriented model for multimodal lung tumor image fusion

Abstract

Multimodal medical image fusion plays an important role in medical clinical applications. However, gradient features and intensity features are not extracted inadequately in fusion methods. To solve the above problems, this paper proposes a Gradient-Intensity oriented Automatic Encode-Decode multimodal lung tumor image fusion model (GIAE-Net), there are two parallel branches in this network, one is gradient branch, and another is intensity branch. The main idea of this proposed network is as follows:Firstly, the gradient attention module (GAM) is designed to enhance the description ability of fine-grained spatial features by using gradient operators, so that the network can retain more edge details. Secondly, the intensity attention module (IAM) is constructed to enable the model to learn image intensity features, which can highlight the lesion region information. Thirdly, the gradient intensity fusion module (GIFM) and the feature flow fusion strategy are designed. It converts the fusion problem into the weight assignment problem of gradient and intensity features, and the image feature extraction and fusion are realized gradually. Finally, a new multimodal medical dataset of lung tumor PET-CT is established, which contains 2575 pairs of PET-CT images (PCLset). The experimental results on PCLset show that compared with other nine fusion models, the proposed model can achieve better fusion performance. In CT lung window images and PET images comparison experiment, SD, IE, AG, QAB/F, VIF and EI indexes are improved by 20.38 %,7.70 %,16.44 %, 21.90 %,11.52 % and 33.95 %, respectively.

Keywords

• Lung Tumor
• Multimodal medical image
• Image fusion
• Gradient feature
• Intensity feature