Deep learning reveals what facial expressions mean to people in different cultures
Jeffrey A. Brooks,
Lauren Kim,
Michael Opara,
Dacher Keltner,
Xia Fang,
Maria Monroy,
Rebecca Corona,
Panagiotis Tzirakis,
Alice Baird,
Jacob Metrick,
Nolawi Taddesse,
Kiflom Zegeye,
Alan S. Cowen
Affiliations
Jeffrey A. Brooks
Research Division, Hume AI, New York, NY 10010, USA; Department of Psychology, University of California, Berkeley, Berkeley, CA 94720, USA; Corresponding author
Lauren Kim
Research Division, Hume AI, New York, NY 10010, USA
Michael Opara
Research Division, Hume AI, New York, NY 10010, USA
Dacher Keltner
Research Division, Hume AI, New York, NY 10010, USA; Department of Psychology, University of California, Berkeley, Berkeley, CA 94720, USA
Xia Fang
Department of Psychology and Behavioral Sciences, Zhejiang University, Hangzhou, Zhejiang, China
Maria Monroy
Department of Psychology, University of California, Berkeley, Berkeley, CA 94720, USA
Rebecca Corona
Department of Psychology, University of California, Berkeley, Berkeley, CA 94720, USA
Panagiotis Tzirakis
Research Division, Hume AI, New York, NY 10010, USA
Alice Baird
Research Division, Hume AI, New York, NY 10010, USA
Jacob Metrick
Research Division, Hume AI, New York, NY 10010, USA
Nolawi Taddesse
Node Survey Solutions, Addis Ababa, Ethiopia
Kiflom Zegeye
Node Survey Solutions, Addis Ababa, Ethiopia
Alan S. Cowen
Research Division, Hume AI, New York, NY 10010, USA; Department of Psychology, University of California, Berkeley, Berkeley, CA 94720, USA; Corresponding author
Summary: Cross-cultural studies of the meaning of facial expressions have largely focused on judgments of small sets of stereotypical images by small numbers of people. Here, we used large-scale data collection and machine learning to map what facial expressions convey in six countries. Using a mimicry paradigm, 5,833 participants formed facial expressions found in 4,659 naturalistic images, resulting in 423,193 participant-generated facial expressions. In their own language, participants also rated each expression in terms of 48 emotions and mental states. A deep neural network tasked with predicting the culture-specific meanings people attributed to facial movements while ignoring physical appearance and context discovered 28 distinct dimensions of facial expression, with 21 dimensions showing strong evidence of universality and the remainder showing varying degrees of cultural specificity. These results capture the underlying dimensions of the meanings of facial expressions within and across cultures in unprecedented detail.
