When words and graphs move the eyes:The processing of multimodal causal relations

Abstract

Research on causal relations in multisemiotic texts constituted by words and graphs has been scarce with only a few exceptions. In the current study, eye movement behavior was studied in seventy-six Chilean high school students, who read a set of twelve causally-related economics texts in Spanish in four experimental conditions. The objective is twofold. We aimed, on the one hand, to observe the main effects of the causal discourse marker (DM) por tanto and the statistical causal graph (G), as well as the interaction effect of both variables on different eye tracking measures. On the other, we seek to observe the effects of the DM on the same eye tracking measures for the graph system (GS) area of interest (AOI). The findings showed that the conjoint presence of the DM and the G did not positively influence the processing of selected AOIs. Analyses also reveal no significant effects on the GS AOI. Thus, the results indicate that the DM tend to decrease processing times, while the G increases them. Additional analyses conducted on the integrative transitions between the verbal system and the graph system reveal that more transitions were identified between the consequence segment and the graph system, thereby confirming that the consequence segment is crucial for the integration of both semiotic systems.

Keywords

• Eye movement
• eye tracking
• causal relations
• discourse marker
• multisemiotic texts
• graphs