Action Recognition and Subsequent In-Depth Analysis for Improving the Time Efficiency of Bimanual Industrial Work

Abstract

Although there are numerous studies that have developed human action recognition (HAR) algorithms, they have mostly focused on accurate recognition of actions; there is a lack of knowledge on analysis and interpretation of the recognition results for identifying the critical factor causing work delay. Further, from a technical standpoint, existing algorithms have difficulty dealing with missing objects during work processes. To overcome these two limitations, this study developed a new HAR algorithm for the recognition of bimanual actions of industrial workers, termed coordinate-BiLSTM with missing object information (C-BiLSTM+MO), and proposed a multi-regression model for conducting in-depth analysis of the recognition results. The proposed HAR algorithm was verified with experimental data from two typical industrial scenarios (pick-and-place, assembly-and-disassembly). The proposed multi-regression model was applied to the recognition results of these tasks and the data from existing bimanual action recognition datasets. The results revealed that the proposed HAR model could recognize the actions of both hands over 85% of the time, for tasks including when an object is missing or appearing, and each key component included in the proposed HAR model could significantly improve the recognition performance. Further, the proposed multi-regression model can explain over 50% of the variance of work time for all seven tasks. Notably, we clarified that the parameter of asymmetricity in the action of the two hands had a significant effect on the work delay for all tasks (p<.01). These results suggest the benefits of in-depth analysis of recognition results to improve time efficiency.

Keywords

• Human action recognition
• human–object interaction
• machine learning
• performance analysis
• time and motion study