Development of a novel robot-assisted vocabulary learning system using pure synthetic data

Abstract

This study investigates the application of deep learning methods trained on synthetic data for the robust detection of vocabulary flashcards, an essential aspect of robot-assisted language learning (RALL). Despite its importance, flashcard detection has not been extensively researched in RALL systems, and it poses significant challenges due to the extensive data collection and annotation needed, especially given the large number of classes involved. These models need to generalize well to different real-world environments. To address these issues, a novel robotic platform designed for flashcard-based vocabulary learning is proposed, supported by a synthetic data generation pipeline using high dynamic range images (HDRIs) and synthetic human actors. The proposed pipeline offers an efficient data generation method and significantly enhances model generalisability to various environments. The proposed method was evaluated with five object detection models based on several challenging real-world datasets, each containing more than 200 class labels. The object detection models trained based on the proposed synthetic datasets (HDRI and HDRI+Humans) demonstrated outstanding performance, achieving median mean average precision (mAP) scores of 0.797 and 0.778. The proposed method achieves this performance without needing to be trained with real data. A comprehensive analysis comparing the proposed method with other synthetic data generation techniques is presented, and its potential for improving vocabulary flashcard detection in RALL systems is highlighted. The data and models are available at https://github.com/aimlab-mu/aimrobot.

Keywords

• Synthetic data generation
• Synthetic-to-real transfer
• Object detection
• Robot assisted language learning
• Human-robot interaction