Ensemble Deep Learning for Classifying Sleeping Posture of Humans Covered in Blankets Using RGB and Thermal Imaging

Abstract

Accurate sleep posture monitoring plays a pivotal role in the diagnosis and treatment of a spectrum of sleep-related disorders, such as sleep apnea, restless leg syndrome, and rapid eye movement sleep behavior disorder. These disorders can significantly affect an individual’s overall well-being and quality of life. However, the complexity of the human body and various factors affecting sleep patterns pose substantial challenges. Thus, this study introduces a sleep position determination approach that utilizes both RGB and thermal cameras to acquire visual and thermal data. This dual-source data acquisition system enables a comprehensive analysis of body positioning during sleep, including potential discomfort assessment. The methodology for this endeavor was initiated by acquiring a dataset encompassing various sleep postures, which was achieved through the deployment of RGB and thermal cameras. This dataset includes video footage of five frequently adopted sleep positions: supine, left log, right log, prone left, and prone right, involving the participation of nine individuals. Furthermore, the dataset was collected under two conditions: with and without a blanket. The proposed method begins by normalizing the database to the video frames. Next, the fine-tuned MobileNet-V2 and Inception-V3 models were employed for feature extraction. The tree-seed algorithm was used to select optimal features from the extracted data, reduce dimensionality, and improve the classification performance. Subsequently, Parallel Standard Deviation Padding Max Value (PSPMV), was applied to combine the feature vectors from the RGB and thermal datasets to enhance the accuracy. The fused vectors are then classified using ensemble machine learning models. Our method achieved an accuracy of 97.8% and 98.4% with and without blanket, using subspace KNN. After applying PSPMV fusion to blanket features, the accuracy improved to 99.2%.

Keywords

• Human sleep posture
• InceptionV3
• MobileNet-V2
• parallel standard deviation padding max value and tree seed algorithm