How important is motion in sign language translation?

Abstract

Abstract More than 70 million people use at least one sign language (SL) as their main channel of communication. Nevertheless, the absence of effective mechanisms to translate massive information among sign, written and spoken languages is the main cause of a negligible inclusion of deaf people into society. Therefore, SL automatic recognition systems have widely proposed to support the characterisation of the sign structure. Today, natural and continuous SL recognition is an open research problem due to multiple spatio‐temporal shape variations, challenging visual sign characterisation, as well as the non‐linear correlation among signs to express a message. A compact sign is introduced to text architecture that explores motion as an alternative to support sign translation. Such characterisation results are robust to appearance variance with relative support to geometrical variations. The proposed representation focuses on the main spatio‐temporal regions to each corresponding word. The proposed architecture was evaluated in a built SL data set (LSCDv1) dedicated to motion study and also in the state‐of‐the‐art RWTH‐Phoenix. From the LSCDv1 data set, the best configuration reports a BLEU‐4 score of 63.04 in a testing set. Regarding RWTH‐Phoenix, the proposed strategy achieved a BLEU‐4 score in a test of 4.56, improving the results under similar reduced conditions.

Keywords

• feature extraction
• handicapped aids
• language translation
• natural language processing
• text analysis
• sign language recognition