Image–Music Synesthesia-Aware Learning Based on Emotional Similarity Recognition

Abstract

Synesthesia is a phenomenon in which human experience a cross-sensory interaction in perception. However, it is hard to bridge two sensory modalities in artificial intelligence. Emotion, the universal content across multiple media modalities, can be a cue to connect sensory perceptions for developing computer-based synesthetic intelligence. In this study, we present an image-music, cross-synesthesia-aware model based on their similarity in the emotion space. In this experiment, we built an affective synesthesia database of 250,000 image-music pairs. Multiple music and image features were extracted to form the database. Emotional representation is abstract and complex in perception, and the recognition of emotional similarity is fraught with uncertainty. In this work, Pearson correlation coefficient (PCC) and Euclidean distance (ED) method was compared to obtain the emotional similarity labels of each affective image-music pair. The proposed method could predict emotional similarity with mean squared error of 0.0075, demonstrating the effectiveness of our approach and may shed light on the development of cross-modal synesthesia-aware systems.

Keywords

• Cross-modal retrieval
• affective computing
• synesthesia-aware modeling
• Pearson correlation coefficient