Image emotion analysis based on semantic concepts

Abstract

With the increasing number of users express their emotions via images on social media, image emotion analysis attracts much attention of researchers. For the ambiguity and subjectivity of emotion, image emotion analysis is more challenging than other computer vision tasks. Previous methods merely learn a direct mapping between image feature and emotion. However, in emotion perception theory of psychology, it is demonstrated that human beings perceive emotion in a stepwise way. Therefore, we propose a novel image emotion analysis framework that makes use of emotional concepts as middle-level feature to bridge image and emotion. Firstly, the relationship between the concept and the emotion is organized in the form of knowledge graph. The relation between the image and the emotion in the semantic embedding space is explored where the knowledge is encoded into. On the other hand, a multi-level deep metric learning method to optimize the model from both label level and instance level is proposed. Extensive experimental results on two image emotion datasets, demonstrate that the present approach performs favorably against the state-of-the-art methods on both affective image retrieval and classification tasks.

Keywords

• 图像情感分析
• 知识图谱
• 视觉-语义嵌入
• 深度度量学习