Multimodal marking of information structure: gesture-prosody alignment across languages

Abstract

In this paper, we first review the existing evidence of gesture-prosody alignment in information structure marking, focusing on specific gestural patterns that were observed to co-occur with various information structure constructions. Then we complement the evidence with the results of a corpus-based study of gesture-speech alignment in Czech. Analyzing a sample of 80 minutes of personal narratives by 16 speakers collected from a Czech multimodal corpus, we observed that by far the most frequent information structure units accompanied by gestures were foci. In line with previous research, we observed that pitch and intensity peaks lag behind the gesture stroke onset (on average by 300 ms). We also provide new evidence for a systematic variation in the duration of the temporal shift related to the marking of discourse contrast.

Keywords

• gesture
• prosody
• information structure
• multimodality
• gesture-speech integration