Frontiers in Neuroscience (Nov 2016)
Interactive Sonification of Spontaneous Movement of Children - Cross-modal Mapping and the Perception of Body Movement Qualities through Sound
Abstract
In this paper we present three studies focusing on the effect of different sound models ininteractive sonification of bodily movement. We hypothesized that a sound model characterizedby continuous smooth sounds would be associated with other movement characteristics thana model characterized by abrupt variation in amplitude and that these associations could bereflected in spontaneous movement characteristics. Three subsequent studies were conductedto investigate the relationship between properties of bodily movement and sound: (1) a motioncapture experiment involving interactive sonification of a group of children spontaneously movingin a room, (2) an experiment involving perceptual ratings of sonified movement data and (3)an experiment involving matching between sonified movements and their visualizations in theform of abstract drawings. In (1) we used a system constituting of 17 IR cameras trackingpassive reflective markers. The head positions in the horizontal plane of 3-4 children weresimultaneously tracked and sonified, producing 3-4 sound sources spatially displayed throughan 8-channel loudspeaker system. We analyzed children’s spontaneous movement in termsof energy-, smoothness- and directness index. Despite large inter-participant variability andgroup-specific effects caused by interaction among children when engaging in the spontaneousmovement task, we found a small but significant effect of sound model. Results from (2) indicatethat different sound models can be rated differently on a set of motion-related perceptual scales(e.g. expressivity and fluidity). Also, results imply that audio-only stimuli can evoke strongerperceived properties of movement (e.g. energetic, impulsive) than stimuli involving both audioand video representations. Findings in (3) suggest that sounds portraying bodily movementcan be represented using abstract drawings in a meaningful way. We argue that the resultsfrom these studies support the existence of a cross-modal mapping of body motion qualitiesfrom bodily movement to sounds. Sound can be translated and understood from bodily motion,conveyed through sound visualizations in the shape of drawings and translated back from sound visualizations to audio. The work underlines the potential of using interactive sonification to communicate high-level features of human movement data.
Keywords
