Exploring Audio Processing in Mixed Reality to Boost Motivation in Piano Learning

Abstract

As technology becomes increasingly important in education, including music, many VR (virtual reality) and AR (augmented reality) applications have been developed to improve skills and knowledge in playing musical instruments, such as piano. However, these applications mainly utilize MIDI input for validating notes. This research explores the potential implementation of mixed reality to enhance piano learning through audio processing capabilities while maintaining users’ motivation. The research used the HoloLens 2 device and the FFT (Fast Fourier Transform) method with peak detection and compared various window functions to determine the most accurate one. Blackman-Harris performed the best, with a 97.28% accuracy rate when tested on complex songs. The application was also tested by 31 participants and evaluated using the HMSAM (Hedonic-Motivation System Adoption Model), revealing that curiosity and joy were the most significant factors influencing the use of the application. The effectiveness of the learning was moderate, with an increase of 31.28%. Although there were limitations in the use of audio processing, it could still be utilized and improved further to keep users motivated to learn piano.

Keywords

• Mixed reality
• digital signal processing
• music
• games