Audiovisual Speech Perception: Acoustic and Visual Phonetic Features Contributing to the McGurk Effect

Abstract

In the best-known example of the McGurk effect, an auditory consonant /b/ that is presented with a face articulating /g/ is heard as a fusion /d/. However, sometimes this kind of stimulus is heard as /g/, ie, a visual-dominant percept. We explored the stimulus features giving rise to these percepts by using two different stimuli at various levels of acoustic noise. The stimulus auditory /apa/ presented with visual /aka/ was most often heard as /aka/ even without noise, and the proportion of visual-dominant percepts increased with noise level. The stimulus auditory/epe/ presented with visual /eke/ was heard mostly as a fusion /ete/, except at the highest noise level, where also /eke/ was heard. The differences in the quality of the McGurk effect were accounted for by the features of the unisensory stimuli. A phonetic analysis showed that the auditory and visual stimulus features were close to those of /t/ in the /e/-context, but not in the /a/-context stimuli. Thus, the type of the McGurk effect—fusion or visual dominance— depended on the quality of the constituent stimulus components, obeying a modality-precision rule: the more reliable the unisensory stimulus, the greater its contribution to the final percept.