IoT-Based Multisensors Fusion for Activity Recognition via Key Features and Hybrid Transfer Learning

Abstract

Human activity recognition (HAR) has attracted significant attention in various fields, including healthcare, smart homes, and human-computer interaction. Accurate HAR can enhance user experience, provide critical health insights, and enable sophisticated context-aware applications. This paper presents a comprehensive system for HAR utilizing both RGB videos and inertial measurement unit (IMU) sensor data. The system employs a multi-stage processing pipeline involving preprocessing, segmentation, feature extraction, and classification to achieve high accuracy in activity recognition. In the preprocessing stage, frames are extracted from RGB videos, and IMU sensor data undergoes denoising. The segmentation phase applies Naive Bayes segmentation for video frames and Hamming windows for sensor data to prepare them for feature extraction. Key features are extracted using techniques such as ORB (Oriented FAST and Rotated BRIEF), MSER (Maximally Stable Extremal Regions), DFT (Discrete Fourier Transform), and KAZE for image data, and LPCC (Linear Predictive Cepstral Coefficients), PSD (Power Spectral Density), AR Coefficient, and entropy for sensor data. Feature fusion is performed using Linear Discriminant Analysis (LDA) to create a unified feature set, which is then classified using ResNet50 (Residual Neural Network) to recognize activities such as using a smartphone, cooking, and reading a newspaper. The system was evaluated using the LARa and HWU-USP datasets, achieving classification accuracies of 92% and 93%, respectively. These results demonstrate the robustness and effectiveness of the proposed HAR system in diverse scenarios.

Keywords

• Human activity recognition (HAR)
• RGB videos
• IMU sensor data
• Naive Bayes segmentation
• recurrent neural network (RNN)
• Shapley additive explanations (SHAP)