Establishing a multimodal dataset for Arabic Sign Language (ArSL) production

Abstract

This paper addresses the potential of Arabic Sign Language (ArSL) recognition systems to facilitate direct communication and enhance social engagement between deaf and non-deaf. Specifically, we focus on the domain of religion to address the lack of accessible religious content for the deaf community. We propose a multimodal architecture framework and develop a novel dataset for ArSL production. The dataset comprises 1950 audio signals with corresponding 131 texts, including words and phrases, and 262 ArSL videos. These videos were recorded by two expert signers and annotated using ELAN based on gloss representation. To evaluate ArSL videos, we employ Cosine similarities and mode distances based on MobileNetV2 and Euclidean distance based on MediaPipe. Additionally, we implement Jac card Similarity to evaluate the gloss representation, resulting in an overall similarity score of 85% between the glosses of the two ArSL videos. The evaluation highlights the complexity of creating an ArSL video corpus and reveals slight differences between the two videos. The findings emphasize the need for careful annotation and representation of ArSL videos to ensure accurate recognition and understanding. Overall, it contributes to bridging the gap in accessible religious content for deaf community by developing a multimodal framework and a comprehensive ArSL dataset.

Keywords

• Arabic Sign Language
• ArSL video corpus
• Audio corpus
• Text corpus
• Gloss
• Friday sermon