A Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder embedded pose for anomaly detection

Abstract

Abstract Due to the robustness of skeleton data to human scale, illumination changes, dynamic camera views, and complex backgrounds, great progress has been made in skeleton‐based video anomaly detection in recent years. The spatio‐temporal graph convolutional network has been proven to be effective in modelling the spatio‐temporal dependencies of non‐Euclidean data such as human skeleton graphs, and the autoencoder based on this basic unit is widely used to model sequence features. However, due to the limitations of the convolution kernel, the model cannot capture the correlation between non‐adjacent joints, and it is difficult to deal with long‐term sequences, resulting in an insufficient understanding of behaviour. To address this issue, this paper applies the Transformer to the human skeleton and proposes the Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder (STEGT‐AE) to improve the capability of modelling. In addition, the multi‐memory model with skip connections is employed to provide different levels of coding features, thereby enhancing the ability of the model to distinguish similar heterogeneous behaviours. Furthermore, the STEGT‐AE has a single encoder‐double decoder architecture, which can improve the detection performance by the combining reconstruction and prediction error. The experimental results show that performances of STEGT‐AE is significantly better than other advanced algorithms on four baseline datasets.

Keywords

• computer vision
• convolutional neural nets
• feature extraction
• pose estimation
• video surveillance