Exploring the Potential of Reconstructed Multispectral Images for Urban Tree Segmentation in Street View Images

Abstract

Deep learning has gained popularity in recent years for reconstructing hyperspectral and multispectral images, offering cost-effective solutions and promising results. Research on hyperspectral image reconstruction feeds deep learning models with images at specific wavelengths and outputs images in other spectral bands. Although encouraging results of previous works, it should be determined to what extent the reconstructed information can lead to an advantage over the captured images. In this context, the present work inspects whether or not reconstructed spectral images add relevant information to segmentation networks for improving urban tree identification. Specifically, we generate red-edge (ReD) and near-infrared (NIR) images from RGB images using a conditional Generative Adversarial Network (cGAN). The training and validation are carried out with 5770 multispectral images obtained after a custom data augmentation process using an urban hyperspectral dataset. The testing outcomes reveal that ReD and NIR can be generated with an average structural similarity index measure of 0.93 and 0.88, respectively. Next, the cGAN generates ReD and NIR information of two RGB-based urban tree datasets (i.e., Jekyll, 3949 samples, and Arbocensus, 317 samples). Subsequently, DeepLabV3 and SegFormer segmentation networks are trained, validated, and tested using RGB, RGB+ReD, and RGB+NIR images from Jekyll and Arbocensus datasets. The experiments show that reconstructed multispectral images might not add information to segmentation networks that enhance their performance. Specifically, the p-values from a T-test show no significant difference between the performance of segmentation networks.

Keywords

• Image to image translation
• multispectral features
• neural networks
• semantic segmentation
• urban trees