TIPA. Travaux interdisciplinaires sur la parole et le langage (Jan 2023)
Multimodalité(s) dans les interactions humaines
Abstract
In our contribution, we will discuss the notion of multimodality in human interactions. First, we will present the different definitions of the concept of multimodality currently used in linguistics. Then, we will focus on some of the current studies on multimodality conducted at the LPL, showing the variety of theoretical and methodological frameworks used as well as the interaction situations considered, ranging from face-to-face to online interactions, from conversations between speakers of the same language to exolingual communication. Finally, in the last part, we will show that despite the variety of approaches, our studies are complementary. We will conclude this chapter by proposing some avenues for future research, particularly based on the phenomenon of interactional failure. 1. The concept of multimodalityInterpersonal communication (Goffman, 1974) is “multimodal” in its broadest sense, if we take into account the research work carried out in linguistics on oral speech (Colletta, 2004), on gestures (Cosnier et al. 1982, McNeill, 1992), in conversational analysis (Mondada, 2005) as well as in social semiotics (Kress & Van Leeuwen, 2001) or in computer-mediated communication (Develotte et al, 2011) and digital discourse analysis (Paveau, 2017). In this first part, we define our main theoretical anchors, one not excluding the other.1.1 Insights from linguistics When we analyze speech in interaction as a multimodal phenomenon, we generally consider three main modalities: the verbal, the vocal (including prosody) and the posturo-mimo-gestural elements (Colletta, 2004). A multimodal linguistic perspective thus takes into account the different modalities and analyses them by putting them in relation. Studying prosody or analysing gesture (or other kinesic aspects such as postures, gazes, and facial mimicry) without the verbal would isolate the modalities and would not show their articulation (Ferré, 2011).1.2 Insights from social semioticsA conception of multimodality through the input of sensory modalities can be complexified if we think that a single modality can be the channel for several semiotic modes. Thus, through the visual modality, it is possible to perceive several semiotic modes: facial expressions, gestures and proxemics among others. Therefore, multimodality can be defined as a characterization of interaction not only in terms of the modalities at work, but also in terms of semiotic modes.In the 1990s, the development of audio-visual and digital technologies led to the emergence of another vision of the concept of multimodality, rooted in the field of social semiotics. The works of Kress and van Leeuwen (2001) have allowed to define new contours of this notion by integrating the different modalities of the communication allowed by the different artifacts. Thus, multimodality is defined as the massive and joint use of various modes of expression (verbal, visual, audio, tactile, etc.) in communication. 1.3 Emerging insights of “digital multimodality”The concept of “digital multimodality” (Wachs & Weber, 2021) is rooted in several fields of research, close to each other but each with its own particularities. The first, founding and precursor field is Computer Mediated Communication (CMC) which has been focused one in online exchanges, mainly in their textual form (Herring, 1996; Anis, 1998), since the beginning of the Internet (and even of the minitel in France). Then come two emerging fields of linguistics in the lineage of CMC: the field of Screen-Based Multimodal Interactions, which has developed in particular around videoconference interactions (Develotte, Kern & Lamy, 2011, Develotte & Paveau, 2017; Guichon & Tellier, 2017) and that of Digital Discourse Analysis (Paveau, 2017). Finally, in the field of language didactics, we will highlight the field of Technology-Enhanced Language Learning (Guichon, 2012) which studies the integration of technologies in communication situations specifically related to language teaching and learning. 2. Research on multimodality at LPLResearch on multimodality at the LPL is divided into three main theoretical frameworks: 1) multimodal speech (verbal, vocal and posturo-mimo-gestural), 2) social semiotics, and finally 3) digital multimodality, and sometimes even a combination of the three. The LPL researchers who are interested in the question of multimodality each study different contexts such as face-to-face interactions, online interactions and hybrid interactions.2.1. Face-to-face interactionsFace-to-face interactions can be of different nature (e.g. conversation, doctor-patient interaction, work meeting…). In this section, we will present some research on face-to-face conversation. Our work focuses on prosody during disfluencies (Pallaud et al., 2019) and on smiling in two phases of conversation: thematic transitions and humor phases (Amoyal et al., 2020). In the study of face-to-face interactions, there is also interest in exolingual communication (between participants who do not have the same first language) (Porquier, 1994). These are particularly interesting to study from a multimodal perspective, especially from the point of view of the adaptation of modalities to facilitate access to meaning (Tellier et al., 2021). 2.2. Online interactionsWith the spread of computers and increasingly powerful Internet connections, several computer-mediated communication (CMC) tools have developed and spread over the years, from the first e-mail exchanges to the recent widespread use of videoconferencing and videophone calls. On the one hand, these new tools have been the object of a projection of interactional processes coming from face-to-face interactions, while on the other hand, new communicative phenomena linked to the multimodality of different forms of CMC have emerged (for example, emoticons in chat rooms). Some research at the LPL has thus turned to the emergence of discursive techno-genres and their characterization, for example in terms of ethos. Others are interested in the pedagogical potential for language didactics, both in terms of second language acquisition and teacher training.2.3. Hybrid interactionsHybrid interactions mix face-to-face and distance learning and can take place in different configurations. An interdisciplinary and collective research between several laboratories, including the LPL (see Présences numériques corpus), studied a poly-artifact doctoral seminar (Develotte et al, 2021), i.e. a seminar where one part of the participants is physically present and the other part is present via different artifacts (a tablet articulated on a base, a human-sized robot mobile on wheels and an interactive multimodal platform) which featured, among other modules, a videoconference space, a chat space, a collective note-taking and a document sharing. Each artifact was itself operated through multiple screens (of computers, tablets, smartphones, etc.). Such a multimodal poly-artifact communication context is therefore eminently complex to use for the participants, as well as to study and transcribe for the researchers. 3. PerspectivesAt the end of this chapter, which highlights the main directions of research carried out at LPL on multimodality, several perspectives are identified. For example, studying the phenomenon of “interactional failure” seems particularly interesting, especially to show how speakers draw on their multimodal resources to overcome these failures. Different phenomena can be observed: disfluencies, lexical searches, misunderstandings and repairs, explanations and finally anticipation of the failure through a multimodal didactic discourse.A first avenue of research that could be explored is the multimodal nature of disfluencies in interaction: is there only a synchronization of the suspension of gesture and speech or are there specific gestures linked to lexical research? A research perspective at LPL could be to compare the occurrence of disfluencies in L1 vs. foreign language speech to see if language proficiency affects how this phenomenon is expressed multimodally. Among the perspectives to be explored in research on multimodality, we can mention the study of conversational alignments. In the model proposed by Pickering and Garrod (2004, 2021), alignment is a psychological phenomenon concerning the mental representations of interlocutors, which is reflected in the articulation and often the repetition of communicative behaviours. While alignment was initially studied through the phenomenon of priming at different language levels (lexicon, morphosyntax, etc.), recently there has been an interest in multimodal communication, including gestures and facial mimicry (Cappellini, Holt and Hsu, 2022). By including the study of facial mimicry, a study of the smiles of interlocutors in the repair phase could highlight the importance of this facial expression in this interactional process. Another avenue of research would be to question the place of the smile in lexical research. Finally, it seems that we could also study the complexity of the interactional scheme of the YouTube platform (a video discourse to which written comments respond in asynchrony) and how vloggers anticipate possible interactional failures by resorting to a didactic discourse (Moirand, 1993) which calls on different multimodal elements.
Keywords
