Bidirectional Braille-speech Communication System for Deafblind Students

Abstract

Deafblindness is a type of dual disability wherein both visual and auditory capabilities are significantly impaired. Special communication methods have been developed for the deafblind community, yet these methods require that both people involved have prior knowledge and training to successfully communicate; limiting deafblind people’s social interactions, particularly in academic settings. This paper describes the development of a device that enables two-way communication between a severe to profoundly deafblind user and a hearing person with no prior knowledge of Braille and no additional intermediaries. A Convolutional Neural Network (CNN) scheme for speech recognition was designed and implemented along with the development of an algorithm capable of developing both text-to-speech and Finger-Braille-to-text conversion. Lastly, a system integration via 3D modelling and additive manufacturing was carried out to deliver a functional prototype. The resulting device aims to allow deafblind students to send and receive information entirely in finger Braille, using buttons and vibrotactile feedback, while the hearing tutor receives auditory messages and speaks to reply, making the educational experience as familiar as possible for both parties. Users testing the device achieved an average typing accuracy of over 95% and demonstrated understanding of commands transmitted through the device’s components.

Keywords

• accessibility
• assistive technology
• braille
• convolutional neural networks
• deafblindness
• education