Multi‐layer fusion techniques using a CNN for multispectral pedestrian detection

Abstract

In this study, a novel multi‐layer fused convolution neural network (MLF‐CNN) is proposed for detecting pedestrians under adverse illumination conditions. Currently, most existing pedestrian detectors are very likely to be stuck under adverse illumination circumstances such as shadows, overexposure, or nighttime. To detect pedestrians under such conditions, the authors apply deep learning for effective fusion of the visible and thermal information in multispectral images. The MLF‐CNN consists of a proposal generation stage and a detection stage. In the first stage, they design an MLF region proposal network and propose to use summation fusion method for integration of the two convolutional layers. This combination can detect pedestrians in different scales, even in adverse illumination. Furthermore, instead of extracting features from a single layer, they extract features from three feature maps and match the scale using the fused ROI pooling layers. This new multiple‐layer fusion technique can significantly reduce the detection miss rate. Extensive evaluations of several challenging datasets well demonstrate that their approach achieves state‐of‐the‐art performance. For example, their method performs 28.62% better than the baseline method and 11.35% better than the well‐known faster R‐CNN halfway fusion method in detection accuracy on KAIST multispectral pedestrian dataset.

Keywords

• multilayer fusion techniques
• CNN
• multispectral pedestrian detection
• multilayer fused convolution neural network
• pedestrian detectors
• adverse illumination circumstances