Pedestrians crossing intention anticipation based on dual‐channel action recognition and hierarchical environmental context

Abstract

Abstract The increase in car ownership has brought convenience to people's lives, but it has also led to a sharp increase in pedestrian–vehicle conflicts. Such conflicts can be avoided by braking when pedestrians in front of the car are detected. However, frequent braking reduces driving comfort and travel efficiency. Considering that pedestrian–vehicle conflicts always occur when pedestrians cross the road, this study proposes a multi‐factor fusion network (MFFN) to anticipate pedestrians' crossing intentions. MFFN contains a dual‐channel action recognition sub‐network to robustly identify pedestrians' actions by adaptively fusing pedestrians' skeleton and representation characteristics. Afterward, object‐level and semantic‐level perception of traffic scenes are realised by a hierarchical graph attention network and a lightweight semantic segmentation method, respectively. These hierarchical environmental contexts can provide critical clues for identifying pedestrians' crossing intentions. Then, a self‐attention mechanism is used to integrate various factors and produce intention identification results. The proposed MFFN is evaluated on public datasets PIE and JAAD. Experimental results show that the proposed method can accurately and reliably identify pedestrian crossing intentions. The accuracy, F1 score, AUC, and precision indicators on JAAD/PIE datasets are 0.912/0.876, 0.813/0.806, 0.896/0.889, and 0.802/0.788, respectively.