SHS Web of Conferences (Jan 2020)

“Pool of Toys”: Multimodal Visual Music Rendering of Sequenced and Real-time Control Events

  • Kawahara Satsuki,
  • Cohen Michael

DOI
https://doi.org/10.1051/shsconf/20207704001
Journal volume & issue
Vol. 77
p. 04001

Abstract

Read online

The goal of this study is to display music visually to aid recognizing each component, enjoying music not just aurally but visually as well. Elements of western music comprise multiple aspects (e.g., rhythm, melody, harmony, and orchestration). To express them, we introduce a “live song” called “Pool of Toys.” A pool filled with toys is used as a motif in a game engine (Unity). A multitimbral song including musical elements was created using DAWs (GarageBand and then Ableton Live). Each part is imported as a MIDI track into Unity, controlling the attributes of visual elements (floating toys) at runtime. A surface grid controller (Launchpad Pro) is used at runtime along with the precomposed music so that performers can synchronously play the MIDI terminal with output displayed in realtime.