Brain‐Controlled Augmented Hearing for Spatially Moving Conversations in Multi‐Talker Environments

Abstract

Abstract Focusing on a specific conversation amidst multiple interfering talkers is challenging, especially for those with hearing loss. Brain‐controlled assistive hearing devices aim to alleviate this problem by enhancing the attended speech based on the listener's neural signals using auditory attention decoding (AAD). Departing from conventional AAD studies that relied on oversimplified scenarios with stationary talkers, a realistic AAD task that involves multiple talkers taking turns as they continuously move in space in background noise is presented. Invasive electroencephalography (iEEG) data are collected from three neurosurgical patients as they focused on one of the two moving conversations. An enhanced brain‐controlled assistive hearing system that combines AAD and a binaural speaker‐independent speech separation model is presented. The separation model unmixes talkers while preserving their spatial location and provides talker trajectories to the neural decoder to improve AAD accuracy. Subjective and objective evaluations show that the proposed system enhances speech intelligibility and facilitates conversation tracking while maintaining spatial cues and voice quality in challenging acoustic environments. This research demonstrates the potential of this approach in real‐world scenarios and marks a significant step toward developing assistive hearing technologies that adapt to the intricate dynamics of everyday auditory experiences.

Keywords

• auditory attention decoding
• brain‐computer interfaces
• intracranial electroencephalography
• hearing aids
• speech separation
• deep learning