Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Abstract

Abstract Convolutional neural networks (CNNs) are deep learning frameworks which are well known for their notable performance in classification tasks. Hence, many skeleton-based action recognition and segmentation (SBARS) algorithms benefit from them in their designs. However, a shortcoming of such applications is the general lack of spatial relationships between the input features in such data types. Besides, non-uniform temporal scalings are a common issue in skeleton-based data streams which leads to having different input sizes even within one specific action category. In this work, we propose a novel deep-aligned convolutional neural network (DACNN) to tackle the above challenges for the particular problem of SBARS. Our network is designed by introducing a new type of filters in the context of CNNs which are trained based on their alignments to the local subsequences in the inputs. These filters result in efficient predictions as well as learning interpretable patterns in the data. Also, our DACNN framework can incrementally expand its deep structure based on the learning progress, which makes it flexible regarding different SBARS datasets. We empirically evaluate our framework on real-world benchmarks showing that the proposed DACNN algorithm obtains a competitive performance compared to the state of the art while benefiting from a less complicated yet more interpretable model.

Keywords

• Convolutional neural networks
• Action recognition
• Time series alignment
• Action segmentation