Journal of King Saud University: Computer and Information Sciences (Sep 2023)

Facilitating the communication with deaf people: Building a largest Saudi sign language dataset

  • Mansour Alsulaiman,
  • Mohammed Faisal,
  • Mohamed Mekhtiche,
  • Mohamed Bencherif,
  • Tariq Alrayes,
  • Ghulam Muhammad,
  • Hassan Mathkour,
  • Wadood Abdul,
  • Yousef Alohali,
  • Mansour Alqahtani,
  • Habib Al-Habib,
  • Hassan Alhalafi,
  • Mohammed Algabri,
  • Muneer Al-hammadi,
  • Hamdi Altaheri,
  • Taha Alfakih

Journal volume & issue
Vol. 35, no. 8
p. 101642

Abstract

Read online

Recently, several countries have been trying hard to facilitate the integration of disabled people into their societies by ensuring equal opportunities through ease of access to social services, daily human necessities, and the labor market. Deafness is considered one of the major disabilities separating the deaf from their society. To integrate the deaf fully into society, a two-way mode of communication is required: one from the deaf to the hearing people, and the other from the hearing to the deaf. Communication from the hearing person to the deaf is generally easy and can be done through speech recognition and text-to-sign representations, but communication from the deaf to the hearing is somewhat difficult and requires a sign recognition module that recognizes the sign motions from the deaf and translates it to a text; following this, a speech synthesis module will translate this text to speech. To build the sign recognition module, a sign language dataset is required. This paper contributes to the literature by introducing a comprehensive survey of 17 Arabic sign language datasets and by developing a well-organized framework that is used to build a sign language dataset. This paper also contributes to the literature by creating the largest Saudi Sign Language (SSL) database—the King Saud University Saudi-SSL (KSU-SSL data-base)—with 293 signs, 33 signers, 145,035 samples, and 10 domains (healthcare, common, alpha-bets, verbs, pronouns and adverbs, numbers, days, kings, family, and regions). This paper also contributes to the literature by introducing a convolutional graph neural network (CGCN) architecture for sign language recognition and applying the proposed architecture to the built KSU-SSL database. The architecture is made up of a small number of separable 3DGCN layers, and is augmented with a spatial attention mechanism. This study is a part of the project that aims to develop a two-way communication system for Saudi deaf individuals.

Keywords