Enhancing Facial Expression Recognition Under Data Uncertainty Based on Embedding Proximity

Abstract

Facial Expression Recognition (FER) on unconstrained datasets poses a significant challenge, primarily due to data uncertainty stemming from human subjectivity and ambiguous facial expressions. Previous methods attempt to address this issue through relabeling strategies. However, this work reveals a relabel inconsistency problem. Specifically, the model weights are not updated simultaneously with the relabeling process. Consequently, the feature representations of the noisy samples remain associated with the previous label despite being relabeled. As a result, the relabeling mechanism reverts the new label back to the previous one, initiating a cycle between the two classes during the subsequent training. The failure to “shift” the feature representations closer to the new label centers hinders the model from learning discriminative features capable of handling data uncertainty, leading to degraded performance. In this work, a new framework based on embedding proximity is proposed to ensure consistent updating of feature representations with rectifications made during relabeling to overcome this limitation. This is achieved by pushing relabeled images closer to their newly assigned class centers and farther away from their previous class (wrong) centers in the feature embedding space. Through comprehensive experiments, this work utilizes existing models-SCN, RUL, and DMUE-to map the original feature space and then applies the proposed embedding proximity technique to update the feature representations. The updated models, denoted as SCN-C, RUL-C, and DMUE-C, demonstrate significant improvements in addressing inconsistency issues and enhancing overall performance. The proposed models outperform state-of-the-art methods, achieving accuracies of 65.73% on AffectNet, 89.51% on RAF-DB, and 71.83% on FER2013.

Keywords

• Data uncertainty
• facial expression recognition
• feature embedding space
• relabeling