Happy emotion cognition of bimodal audiovisual stimuli optimizes the performance of the P300 speller

Abstract

Abstract Objective Prior studies of emotional cognition have found that emotion‐based bimodal face and voice stimuli can elicit larger event‐related potential (ERP) amplitudes and enhance neural responses compared with visual‐only emotional face stimuli. Recent studies on brain–computer interface have shown that emotional face stimuli have significantly improved the performance of the traditional P300 speller system, but its performance needs to be further improved for practical applications. Therefore, we herein propose a novel audiovisual P300 speller based on bimodal emotional cognition to further improve the performance of the P300 system. Methods The audiovisual P300 speller we proposed is based on happy emotions, with visual and auditory stimuli that consist of several pairs of smiling faces and audible chuckles (E‐AV spelling paradigm) of different ages and sexes. The control paradigm was the visual‐only emotional face P300 speller (E‐V spelling paradigm). Results We compared the ERP amplitudes, accuracy, and raw bit rate between the E‐AV and E‐V spelling paradigms. The target stimuli elicited significantly increased P300 amplitudes (p < .05) and P600 amplitudes (p < .05) in the E‐AV spelling paradigm compared with those in the E‐V paradigm. The E‐AV spelling paradigm also significantly improved the spelling accuracy and the raw bit rate compared with those in the E‐V paradigm at one superposition (p < .05) and at two superpositions (p < .05). Significance The proposed emotion‐based audiovisual spelling paradigm not only significantly improves the performance of the P300 speller, but also provides a basis for the development of various bimodal P300 speller systems, which is a step forward in the clinical application of brain–computer interfaces.

Keywords

• audiovisual
• brain–computer interface
• emotional cognition
• face
• P300 speller
• voice