The Impact of Hill-Type Actuator Components on the Performance of Reinforcement Learning Controllers to Reverse Upper-Limb Paralysis

Abstract

Functional electrical stimulation (FES) may allow people who are paralyzed due to spinal cord injuries (SCIs) to regain the ability to move. Deep neural networks (DNNs) trained with reinforcement learning (RL) have been recently explored as a promising methodology to control FES systems to restore upper-limb movements. However, previous studies suggested that large asymmetries in antagonistic upper-limb muscle strengths could impair RL controller performance. In this work, we investigated the underlying causes of asymmetry-associated decreases in controller performance by comparing different Hill-type models of muscle atrophy, and by characterizing RL controller sensitivity to passive mechanical properties of the arm. Simulations indicated that RL controller performance is relatively insensitive to moderate (up to 50%) changes in tendon stiffness and in flexor muscle stiffness. However, the viable workspace for RL control was substantially affected by flexor muscle weakness and by extensor muscle stiffness. Furthermore, we uncovered that RL controller performance issues previously attributed to asymmetrical antagonistic muscle strength resulted from flexor muscle active forces that were insufficient to counteract extensor muscle passive resistance. The simulations supported the adoption of rehabilitation protocols for reaching tasks that prioritize decreasing muscle passive resistance, and counteracting passive resistance with increased antagonistic muscle strength.

Keywords

• Reinforcement learning
• functional electrical stimulation
• motor control
• muscle atrophy
• musculoskeletal model