Spatio‐temporal multi‐scale motion descriptor from a spatially‐constrained decomposition for online action recognition

Abstract

This study presents a spatio‐temporal motion descriptor that is computed from a spatially‐constrained decomposition and applied to online classification and recognition of human activities. The method starts by computing a dense optical flow without explicit spatial regularisation. Potential human actions are detected at each frame as spatially consistent moving regions of interest (RoIs). Each of these RoIs is then sequentially partitioned to obtain a spatial representation of small overlapped subregions with different sizes. Each of these region parts is characterised by a set of flow orientation histograms. A particular RoI is then described along the time by a set of recursively calculated statistics that collect information from the temporal history of orientation histograms, to form the action descriptor. At any time, the whole descriptor can be extracted and labelled by a previously trained support vector machine. The method was evaluated using three different public datasets: (i) the ViSOR dataset was used for global classification obtaining an average accuracy of 95% and for recognition in long sequences, achieving an average per‐frame accuracy of 92.3%. (ii) The KTH dataset was used for global classification and (iii) the UT‐datasets were used for recognition task, obtaining an average accuracy of 80% (frame rate).

Keywords

• spatio-temporal multiscale motion descriptor
• spatially-constrained decomposition
• online action recognition
• human activity online classification
• multiscale dense optical flow
• regions of interest