A Design of Smart Beaker Structure and Interaction Paradigm Based on Multimodal Fusion Understanding

Abstract

Virtual chemistry experiment is an important teaching tool in middle schools. It can not only help students understand the experimental principles, but also help them memorize the experimental procedures. However, there are still some problems in virtual chemistry experiments. First, in existing research, user intentions are often misunderstood and cannot be accurately understood. Second, the existing research fail to identify the user's wrong actions, which reduce the accuracy of the experiment. Third, the user sense of operation and realism is not strong during the experiment, which reduces the user's experience. Finally, the lack of navigation guidance for experimental operations increases user learning time. In order to solve these problems. This paper proposes a scheme for establishing an intelligent navigational chemical laboratory based on multimodal fusion. First of all, we design a new smart beaker structure with perceptual ability, which can be used to complete most chemical experiments and give users a real sense of experience. Besides, we propose a multimodal fusion understanding algorithm, which reduces the misidentification of the experiment and better understands the real intention of the user. Finally, intelligent navigation and wrong behavior recognition functions are added to the experimental equipment, which improves the efficiency of human-computer interaction. The results show that Compared with the existing virtual laboratory or system, the chemical laboratory scheme proposed in this paper through multimodal fusion understanding algorithm greatly reduces the user's memory load and improves the success rate of the experiment. Moreover, through the combination of virtual and real, the virtual chemistry experiment not only improves the authenticity of the operation, but also stimulates the students' interest in learning, which is well received by users.

Keywords

• Human–computer interaction
• multimodal fusion
• intention understanding
• smart beaker
• navigation interaction paradigm