FFUNet: A novel feature fusion makes strong decoder for medical image segmentation

Abstract

Abstract Convolutional neural networks (CNNs) have strong ability to extract local features, but it is slightly lacking in extracting global contexts. In contrast, transformers are good at long‐distance modelling due to the global self‐attention mechanisms while its performance in localization is limited. On the other hand, the feature gap between an encoder and decoder is also challenging for a U‐shaped network, which adopts a plain skip connection. Inherited from convolutional networks and transformers, FFUNet, a hybrid network structure with a novel module named the Feature Fusion Module (FFM) is proposed for medical image segmentation. The proposed FFM consisting of Feature Attention Selection, Cross Offset Generation and Deformable Convolution Layer, aims to replace the original plain skip connection to alleviate the ambiguous semantic information between the encoder and decoder for a more powerful medical image segmentation network. Extensive experiments demonstrate that the proposed FFUNet has amazing performance in segmentation gains on the Synapse dataset. In addition, consistent improvements are also achieved across other four popular datasets and CNN‐based or transformer‐based segmentation networks, which illustrate that the proposed method has advantages in generalization and compactness.

Keywords

• attention mechanism
• conformer
• deformable convolution
• feature fusion
• medical image segmentation
• skip connection