Spatio-Spectral Deep Color Constancy With Multi-Band NIR

Abstract

This paper proposes to utilize deep spatio-spectral features for color constancy, while most conventional methods focus on only spatial information of RGB images. We propose a novel color constancy method that exploits multi-band near-infrared (NIR). The proposed illuminant estimation network comprises two subnetworks, each tasked with learning different aspects: one focuses on capturing the local illuminant based on the input RGB and NIR images, while the other is dedicated to estimating confidence levels using both RGB and NIR inputs. We leverage a cross-attention mechanism to effectively capture the correlation between RGB and NIR modalities. Additionally, our method capitalizes on the relationship between RGB and NIR by incorporating a NIR colorization network into the primary illuminant estimation framework. Due to the absence of a public dataset for color constancy with multi-band NIR, we constructed a dataset comprising 604 images using a hyperspectral camera. The experimental results demonstrate superior performance in illuminant estimation when utilizing multi-band NIR compared to conventional RGB and single-band NIR approaches. Furthermore, employing a cross-attention mechanism to consider the correlation between RGB and NIR reveals the estimation of a sophisticated confidence map that aids in illuminant estimation, surpassing other existing fusion methods. The feature transfer from an additional task network can further enhance the performance of illuminant estimation.

Keywords

• Color constancy
• multi-band near-infrared (NIR)
• illuminant estimation
• spatio-spectral