Interactive and audience-adaptive information interfaces

Abstract

In the doctoral thesis we developed an interactive and user-adaptive information interface based on computer vision and machine learning methods. By using a camera-enhanced digital signage display we employed real-time computer vision algorithms to extract temporal, spatial, and demographic features of the observers, which are further used for observer specific broadcasting of digital signage contents. The algorithms were chosen and modified to optimize the balance between accuracy and time complexity, subjected to design-aim to perform in real-time and using conventional hardware. More particularly, we used the Mixture of Gaussians method for background segmentation, Viola & Jones method for face detection algorithm, Active Appearance Models for face alignment and POSIT algorithm for head pose estimation. The developed interface is used as the key research tool to explore three currently open problems in the field of human-computer interaction: dynamic anamorphosis, quantitative audience measurement study of digital signage in real-world environment, and modeling of the purchase decision process. In the first study, we developed a new interactive computer vision based method which adapts image projection to the changing position of the observer so that wherever the observer moves, he sees the same undeformed image. We call this capacity dynamic anamorphosis. We formalized the anamorphic transformation and proposed a real-time algorithm for tracking the 3D position of the observer's eyes and the re-computation of the anamorphic deformation. As an interesting application, we show that dynamic anamorphosis could be used to improve eye-contact in videoconferencing. In the second study, we used the developed interface to perform a quantitative audience measurement field study, which evaluates user attention. Temporal metrics of a person's dwell time, display in-view time and attention time are extracted using real-time image analysis. The system also determines demographic metrics of the gender and age group based on images of faces. The digital signage display was deployed in a real-world environment of a clothing boutique, where demographic and viewership data of 1294 store customers were recorded, manually verified and analysed. The analysis shows that 35% of customers specifically looked-at the display, having the average attention time of 0.7 s. Interestingly, the attention time was substantially higher for men (1.2 s) than for women (0.4 s). In the third study, the interface is applied to model the purchase decision process, which is an interdisciplinary study, where data collected with the developed interface and subjected to machine learning are combined to model and analyze the decision and roles in a purchasing process. Finally, more generally, the developed system presents a contribution to the field of human-computer interaction and shows further possibilities for scientific use and applications, such as open problem of display blindness, development of new interactive methods for broadcasting of relevant content, and quantitative analysis of user behavior.

Keywords

• Computer Vision
• Video Analysis
• Machine Learning and Data Mining
• Multimodal Interaction and Multimedia
• Applications