Prior Exposure to Instructional Manipulation Checks does not Attenuate Survey Context Effects Driven by Satisficing or Gricean Norms

Abstract

Instructional manipulation checks (IMCs) are frequently included in unsupervised online surveys and experiments to assess whether participants pay close attention to the questions. However, IMCs are more than mere measures of attention – they also change how participants approach subsequent tasks, increasing attention and systematic reasoning. We test whether these previously documented changes in information processing moderate the emergence of response effects in surveys by presenting an IMC either before or after questions known to produce classic survey context effects. When the items precede an IMC, familiar satisficing as well as conversational effects replicate. More important, their pattern and size does not change when the items follow an IMC, in contrast to experiments with reasoning tasks. Given a power of 82% to 98% to detect an effect of d = .3, we conclude that prior exposure to an IMC is unlikely to increase or attenuate these types of context effects in surveys.

Keywords

• instructional manipulation checks
• survey context checks
• satisficing
• Gricean conversational norms
• survey methods