Offline Evaluation Matters: Investigation of the Influence of Offline Performance on Real-Time Operation of Electromyography-Based Neural-Machine Interfaces

Abstract

There has been a debate on the most appropriate way to evaluate electromyography (EMG)-based neural-machine interfaces (NMIs). Accordingly, this study examined whether a relationship between offline kinematic predictive accuracy (R2) and user real-time task performance while using the interface could be identified. A virtual posture-matching task was developed to evaluate motion capture-based control and myoelectric control with artificial neural networks (ANNs) trained to low (R2 ≈ 0.4), moderate (R2 ≈ 0.6), and high ( $\text {R}^{\vphantom {\text {D}^{\text {a}}}{2}} \approx 0.8$ ) offline performance levels. Twelve non-disabled subjects trained with each offline performance level decoder before evaluating final real-time posture matching performance. Moderate to strong relationships were detected between offline performance and all real-time task performance metrics: task completion percentage (r = 0.66, p < 0.001), normalized task completion time (r = −0.51, p = 0.001), path efficiency (r = 0.74, p < 0.001), and target overshoots (r = −0.79, p < 0.001). Significant improvements in each real-time task evaluation metric were also observed between the different offline performance levels. Additionally, subjects rated myoelectric controllers with higher offline performance more favorably. The results of this study support the use and validity of offline analyses for optimization of NMIs in myoelectric control research and development.

Keywords

• Electromyography
• machine learning
• neural–machine interface
• offline analysis
• real-time performance