Video‐based action recognition using spurious‐3D residual attention networks

Abstract

Abstract Recently, 3D Convolutional Neural Networks (3D CNNs) have attracted extensive attention in extracting spatial and temporal features in videos for their efficient feature extraction ability. However, it also brings enormous model parameters by training very deep 3D CNNs. Here, a novel network named spurious‐3D Residual Attention Networks (S3D RANs) is proposed for video‐based action recognition, which has the powerful capacity to learn collaborative spatiotemporal features. In particular, by leveraging the merits from 2D Convolutional Neural Networks (2D CNNs) and 3D CNNs, 2D CNNs are applied rather than 3D CNNs on frames of the single view of volumetric videos data to learn temporal motion features directly. Furthermore, view and channel‐wise attention mechanism submodules are employed in the residual unit to learn the importance of each view for action recognition and guide the network to pay more attention to the more useful information for action recognition. Experimental results on UCF‐101, HMDB‐51 datasets demonstrate that our S3D RANs have higher accuracy and lower model complexity than existing works.