How Subtle Protocol Choices Can Affect Biological Conclusions: Great Tits' Response to Allopatric Mobbing Calls

Abstract

In the last ten years, numerous replicated studies showed divergent results from the original papers, leading to the recognition that science may be facing a replication crisis. Apart from fraud or natural population variability, different results may emerge from flexibility in the protocol and/or restricted sample sizes. Replication studies are therefore fundamental to assess how robust a finding can be. However, while the original authors may be prone to p-hacking (to collect data, select data or use statistical analyses until nonsignificant results become significant), the replication-authors are, on the contrary, probably unwittingly prone to show opposite results (i.e., null-hacking). In this study, we face the unique opportunity to compare replicated studies with no null-hacking bias. Two teams independently investigated the response of great tits (Parus major) to mobbing calls of an allopatric species, in their natural and reversed order, on the same population. We first discuss how subtle protocol choices, especially regarding the soundtrack preparation and playback methodology, can explain variation in the results. In addition, we show that, although the effect sizes of the differences of interest are similar, biological conclusions solely based on the p-value would be different. Thus, we note the pitfall of reliance on p values, especially with small samples.

Keywords

• animal communication
• heterospecific communication
• mobbing
• replication crisis
• syntax