AfriSign: African sign languages machine translation

Abstract

Abstract Research on sign language translation is ongoing with a high social inclusive goal of crossing the bridge between people with hearing disability using sign language as their basic way to communicate to others who do not understand sign language. Hundreds of different sign languages exist instead of a single universal sign language. Research on translating sign languages from high-income nations has grown significantly, but little is known about translating sign languages from Africa. In this paper, we curate a novel video-to-text African sign languages translation dataset containing sign language videos of Bible verses from six (6) different African countries. We experimented with competitive machine translation and sign language translation techniques on our dataset, including the application of transformers to sign language translation, multilingual training, and cross-transfer learning. We evaluated them in terms of accuracy and precision. The results from our experiments prove that having one Multilingual model for all the languages tends to be a better choice when deployed in real system in terms of memory usage with an accuracy of 94.6% and precision of 97.3%. These results give headway for more multilingual models to be developed to enhance inclusion for the deaf community and bridge the gap between the hearing and the deaf in Africa.

Keywords

• Sign language translation
• African sign languages
• Cross transfer learning
• Hearing disability
• Multilingual training
• Competitive machine translation