High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet

Abstract

To solve the problem of poor quality in ghost imaging via sparsity constraints (GISC) multispectral image reconstruction with correlation operations and compressed sensing algorithms under low sampling rate detection conditions, we propose an end-to-end deep-learning-based method. Based on the U-Net, Res2Net-SE-Conv is employed instead of convolutional blocks to extract local and global image features at a more fine-grained level while adaptively adjusting the channel feature response. The two-dimensional contextual transformer is constructed to fully use contextual correlation information to enhance the effectiveness of feature representations. We employ the two-dimensional contextual transformer in the decoder part, dubbed CoT-Unet, to reconstruct the desired 3D cube. The results show that compared with U-Net, TSA-Net based on spatial-spectral self-attention, the PSNR of reconstructed images by the CoT-Unet is improved by 5 dB and 3 dB, respectively, SSIM is improved by 0.23 and 0.07, and SAM is decreased by 0.06 and 0.58. Compared with conventional algorithms such as DGI and CS, our method significantly improves the quality of reconstructed images. Furthermore, the comparison results at 10%, 20%, and 30% sampling rates show that our approach has the best quality in reconstructing GISC multispectral images at low sampling rates.

Keywords

• Multispectral image reconstruction
• convoluti onal neural network
• transformer
• self-attention mechanism
• ghost imaging