Inter-Dimensional Correlations Aggregated Attention Network for Action Recognition

Abstract

Recently, 3D Convolutional Neural Network (3D-CNN) models with attention mechanisms have been widely studied in action recognition tasks. Although most of these methods explore the spatial, temporal and channel attention for action recognition, the inter-correlations over spatial, temporal and channel are not fully exploited. In this paper, we introduce a novel inter-dimensional correlations aggregated attention (ICAA) network that extracts inter-correlations between two dimensions in spatial, temporal and channel, and inter-spatial-temporal-channel correlations to obtain more comprehensive correlations. The proposed ICAA module can be wrapped as a generic module easily plugged into the state-of-the-art 3D-CNN models as well as multi-stream architectures for video action recognition. We extensively evaluate our method on action recognition tasks over UCF-101 and HMDB-51 datasets, and the experimental results demonstrate that adding our ICAA module can obtain state-of-the-art performance on UCF-101 and HMDB-51, which has the performance of 98.4% and 81.9% respectively, and achieve significant improvement compared against original models.

Keywords

• Action recognition
• 3D-CNN
• attention mechanism