Lightweight Long and Short-Range Spatial-Temporal Graph Convolutional Network for Skeleton-Based Action Recognition

Abstract

In skeleton-based human action recognition domain, the methods based on graph convolution networks have great success recently. However, most graphical neural networks consider the skeleton as a spatiotemporally uncorrelated graph and rely on a predetermined adjacency matrix, ignoring the spatiotemporal relevance of human actions and taking up significant computational costs. Meanwhile, the methods use graph convolution to focus too much on the neighboring nodes of the joints and ignore the totality of the action. In this work, we propose a lightweight but efficient neural network called NLB-ACSE based on the Graph Convolutional Network (GCN). Our model consists of two large branches: non-local block branch that focuses on the long distance features and adaptive cross-spacetime edge branch that focuses on the short distance features. Both branches extract information across time and space, and focus on long and short information. Some simple but effective strategies also are applied to our model, such as semantics, maxpooling, and fusion inputs, which have small parameter burden but obtain a higher accuracy on ablation study. The proposed method with an order of magnitude smaller size than most previous papers is evaluated on three large datasets, NTU60, NTU120, and Northwesten-UCLA. The experimental results show that our method achieves the state-of-the-art performance.

Keywords

• Skeleton-based action recognition
• graph convolution network
• fusion inputs
• non-local block
• adaptive cross-spacetime edges