Multimodal Interaction and AI in Rehabilitation of Rheumatoid Arthritis

Abstract

Currently, there are several devices used in kinetotherapy for rehabilitation and evaluation of movements, but fewer applications that support rehabilitation for people diagnosed with rheumatoid arthritis. In the past several ones were using Microsoft Kinect, other are using Leap Motion and artificial intelligence, and present gamification features. The paper presents a system for at-home rehabilitation of patients with first and second stages of rheumatoid arthritis (RA) based on multimodal interaction using leap motion, serious gaming and neuronal networks support. The system consists of an application for the doctor - who will give the diagnostic, who can view the actual patients and the deleted ones - and one for the kinetotherapist with two games matching the symptoms for first and second stage of RA. The aim of the game for RA first stage is to increase overall hand mobility through the swipe movement. The purpose of the second game is to recover the grip movement of the hand, placing some 3D models in a box. Through the neuronal network the patients can have feedback from the comfort of their home for the realized exercises. The correct movements are classified with an accuracy of 95%. The technologies used were: Visual Studio 2019, Unity 2018, C# and Python 3.7. In this moment, the application was tested by a group of 10 patients from Medical Centre Sf. Mary of Timişoara from May to June 2021. The fatigue of the fingers and wrist were, in most of the cases small, respectively, too small. Most of the users given positive feedback and confirmed that they would use the application with the aim of rehabilitation. In the current pandemic context such a system would be very useful for both: patients and healthcare workers. In this way, results would be more visible in terms of rehabilitation.

Keywords

• virtual reality
• hand rehabilitation
• leap motion
• neuronal network
• multimodal interaction