Multimodal Daily-Life Emotional Recognition Using Heart Rate and Speech Data From Wearables

Abstract

Human emotion plays a significant role in mental well-being, and recognizing these emotions in daily life is essential. With the advancement of artificial intelligence, affective computing has paved the way for effective applications in enhancing emotional states in everyday life. In practical daily-life scenarios, data sources that can be collected through simple and low-cost wearables contribute to recognizing emotions in daily routines. Heart rate and speech data can be easily collected from affordable smartwatches without any other human intervention. Heart rate data, directly correlated with physiological response, and speech data, with its rich expressiveness, together yield a robust indicator of the human’s emotional condition. We conduct multimodal emotion recognition (MER), integrating heart rate-based emotion recognition (HER) and speech-based emotion recognition (SER) through a score-based fusion method. Our proposed MER achieves an overall accuracy of 84.22%, surpassing single-modality models with accuracies of 57.65% for HER and 80.38% for SER. The findings highlight the practicality of utilizing emotion-related data sources collected conveniently with smartwatches, thereby enhancing emotion tracking accessibility in daily life scenarios. Furthermore, integrating these modalities proves more effective in capturing emotions than using a single modality. Moreover, our system’s lightweight architecture facilitates easy expansion to incorporate additional modalities, ensuring durable precision even when not all modalities are sensed, making it versatile and pragmatic.

Keywords

• Affective computing
• deep learning
• emotion recognition
• fusion
• multimodal