Vision-Based Hand Rotation Recognition Technique with Ground-Truth Dataset

Abstract

The existing question-and-answer screening test has a limitation in that test accuracy varies due to a high learning effect and based on the inspector’s competency, which can have consequences for rapid-onset cognitive-related diseases. To solve this problem, a behavioral-data-based screening test is necessary, and there are various types of tasks that can be adopted from previous studies, or new ones can be explored. In this study, we came up with a continuous hand movement, developed a behavioral measurement technology, and conducted validity verification. As a result of analyzing factors that hinder measurement accuracy, this measurement technology used a web camera to measure behavioral data of hand movements in order to lower psychological barriers and to pose no physical risk to subjects. The measured hand motion is a hand rotation that repeatedly performs an action in which the bottom of the hand is seen in front. The number of rotations, rotation angle, and rotation time generated by the hand rotation are derived as measurements; and for calculation, we performed hand recognition (MediaPipe), joint data detection, motion recognition, and motion analysis. To establish the validity of the derived measurements, we conducted a verification experiment by constructing our own ground-truth dataset. The dataset was developed using a robot arm with two-axis degrees of freedom and that quantitatively controls the number, time, and angle of rotations. The dataset includes 540 data points comprising 30 right- and left-handed tasks performed three times each at distances of 57, 77, and 97 cm from the camera. Thus, the accuracy of the number of rotations is 99.21%, the accuracy of the rotation angle is 91.90%, and the accuracy of the rotation time is 68.53%, making the overall rotation measurements more than 90% accurate for input data at 30 FPS for measuring the rotation time. This study is significant in that it not only contributes to the development of technology that can measure new behavioral data in health care but also shares image data and label values that perform quantitative hand movements in the image processing field.

Keywords

• cognitive screening
• hand movement detection
• image processing