The CABB dataset: A multimodal corpus of communicative interactions for behavioural and neural analyses
Lotte Eijk,
Marlou Rasenberg,
Flavia Arnese,
Mark Blokpoel,
Mark Dingemanse,
Christian F. Doeller,
Mirjam Ernestus,
Judith Holler,
Branka Milivojevic,
Asli Özyürek,
Wim Pouw,
Iris van Rooij,
Herbert Schriefers,
Ivan Toni,
James Trujillo,
Sara Bögels
Affiliations
Lotte Eijk
Centre for Language Studies, Radboud University, Nijmegen, the Netherlands
Marlou Rasenberg
Centre for Language Studies, Radboud University, Nijmegen, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands
Flavia Arnese
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
Mark Blokpoel
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
Mark Dingemanse
Centre for Language Studies, Radboud University, Nijmegen, the Netherlands
Christian F. Doeller
Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; Kavli Institute for Systems Neuroscience, Centre for Neural Computation, The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, Jebsen Centre for Alzheimer's Disease, Norwegian University of Science and Technology, Trondheim, Norway; Wilhelm Wundt Institute of Psychology, Leipzig University, Leipzig, Germany
Mirjam Ernestus
Centre for Language Studies, Radboud University, Nijmegen, the Netherlands
Judith Holler
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands
Branka Milivojevic
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
Asli Özyürek
Centre for Language Studies, Radboud University, Nijmegen, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands; Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
Wim Pouw
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands
Iris van Rooij
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Department of Linguistics, Cognitive Science, and Semiotics, and the Interacting Minds Centre at Aarhus University, Denmark
Herbert Schriefers
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
Ivan Toni
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
James Trujillo
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands
Sara Bögels
Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Department of Cognition and Communication, Tilburg University, the Netherlands; Corresponding author at: Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands.
We present a dataset of behavioural and fMRI observations acquired in the context of humans involved in multimodal referential communication. The dataset contains audio/video and motion-tracking recordings of face-to-face, task-based communicative interactions in Dutch, as well as behavioural and neural correlates of participants’ representations of dialogue referents. Seventy-one pairs of unacquainted participants performed two interleaved interactional tasks in which they described and located 16 novel geometrical objects (i.e., Fribbles) yielding spontaneous interactions of about one hour. We share high-quality video (from three cameras), audio (from head-mounted microphones), and motion-tracking (Kinect) data, as well as speech transcripts of the interactions. Before and after engaging in the face-to-face communicative interactions, participants’ individual representations of the 16 Fribbles were estimated. Behaviourally, participants provided a written description (one to three words) for each Fribble and positioned them along 29 independent conceptual dimensions (e.g., rounded, human, audible). Neurally, fMRI signal evoked by each Fribble was measured during a one-back working-memory task. To enable functional hyperalignment across participants, the dataset also includes fMRI measurements obtained during visual presentation of eight animated movies (35 min total). We present analyses for the various types of data demonstrating their quality and consistency with earlier research. Besides high-resolution multimodal interactional data, this dataset includes different correlates of communicative referents, obtained before and after face-to-face dialogue, allowing for novel investigations into the relation between communicative behaviours and the representational space shared by communicators. This unique combination of data can be used for research in neuroscience, psychology, linguistics, and beyond.
