Audition controls the flow of visual time during multisensory perception
Mariel G. Gonzales,
Kristina C. Backer,
Yueqi Yan,
Lee M. Miller,
Heather Bortfeld,
Antoine J. Shahin
Affiliations
Mariel G. Gonzales
Department of Cognitive and Information Sciences, University of California, 5200 N Lake Rd, Merced, CA 95343, USA
Kristina C. Backer
Department of Cognitive and Information Sciences, University of California, 5200 N Lake Rd, Merced, CA 95343, USA; Health Sciences Research Institute, University of California, 5200 N Lake Rd, Merced, CA 95343, USA
Yueqi Yan
Health Sciences Research Institute, University of California, 5200 N Lake Rd, Merced, CA 95343, USA
Lee M. Miller
Center for Mind and Brain, University of California, Davis, CA 95618, USA; Department of Neurobiology, Physiology and Behavior, University of California, Davis, CA 95616, USA; Department of Otolaryngology / Head & Neck Surgery, University of California, Davis, Sacramento, CA, 95817, USA
Heather Bortfeld
Department of Cognitive and Information Sciences, University of California, 5200 N Lake Rd, Merced, CA 95343, USA; Health Sciences Research Institute, University of California, 5200 N Lake Rd, Merced, CA 95343, USA; Department of Psychological Sciences, University of California, Merced, CA 95343, USA
Antoine J. Shahin
Department of Cognitive and Information Sciences, University of California, 5200 N Lake Rd, Merced, CA 95343, USA; Health Sciences Research Institute, University of California, 5200 N Lake Rd, Merced, CA 95343, USA; Corresponding author
Summary: Previous work addressing the influence of audition on visual perception has mainly been assessed using non-speech stimuli. Herein, we introduce the Audiovisual Time-Flow Illusion in spoken language, underscoring the role of audition in multisensory processing. When brief pauses were inserted into or brief portions were removed from an acoustic speech stream, individuals perceived the corresponding visual speech as “pausing” or “skipping”, respectively—even though the visual stimulus was intact. When the stimulus manipulation was reversed—brief pauses were inserted into, or brief portions were removed from the visual speech stream—individuals failed to perceive the illusion in the corresponding intact auditory stream. Our findings demonstrate that in the context of spoken language, people continually realign the pace of their visual perception based on that of the auditory input. In short, the auditory modality sets the pace of the visual modality during audiovisual speech processing.
