Multispectral pedestrian detection based on feature complementation and enhancement

Abstract

Abstract Multispectral pedestrian detection with visible light and infrared images is robust to changes in lighting conditions and therefore is of great importance to numerous applications that require all‐day environmental perception. This paper proposes a novel method named FCE‐RCNN, which integrates saliency detection as a sub‐task and utilizes global information for enhanced feature representation. The approach enhances thermal inputs by incorporating gradients at the raw‐data level before feature extraction. Utilizing a dual‐stream backbone, a global semantic information extraction module is introduced that combines pooling with horizontal–vertical attention mechanisms, capturing high‐quality global semantic information for lower‐level feature enrichment and guidance. Additionally, the pedestrian locality enhancement module is designed to enhance spatial locality information of pedestrians through saliency detection. Furthermore, to alleviate the challenges posed by positional shifts between cross‐spectral features, deformable convolution is innovatively employed. Experimental results on the KAIST dataset demonstrate that FCE‐RCNN significantly improves nighttime detection, achieving a log‐average miss rate of 6.92%, outperforming the new method ICAFusion by 0.93%. These results underscore the effectiveness of FCE‐RCNN, and the method also maintains competitive inference speed, making it suitable for real‐time applications.

Keywords

• automated driving
• intelligent vehicles
• computer vision
• image fusion
• infrared imaging