Real-Time Human Action Anomaly Detection Through Two-Stream Spatial-Temporal Networks

Abstract

Human Action Anomaly Detection is an advanced technology that leverages computer vision and machine learning to identify unusual or suspicious human activities in real-time video stream and sensor data. By continuously monitoring environments such as public spaces, workplaces, and residential areas, this system can promptly detect and respond to potential threats and safety violations. The core components of this technology include data collection from video surveillance and sensors, preprocessing techniques for feature extraction, model training using normal action patterns, and real-time anomaly detection algorithms. Due to the hardware limitations in many industrial scenarios, we would like to explore the use of the CPU alone to detect abnormal human action. In this paper, we present a two-stream spatial-temporal transformer network for predicting human-object interactions in real-time. Initially, human skeleton information was extracted using a human pose detector. A human hand detector and tracker are utilized to localize the human hand and detect its joints in greater detail. Concurrently, an object detector was employed to spatially and semantically localize and classify the objects with which the worker in the image was interacting. With extracted information as input, we model human action anomaly detection in operations using both spatial and temporal dimensions. We validated our work with the composed network on our own collected dataset and experimentally proved that our work can significantly identify incorrect work steps in a work scenario.

Keywords

• Anomaly detection
• human-object interaction
• action recognition
• deep learning
• real-world infrastructure