Can peer review accolade awards motivate reviewers? A large-scale quasi-natural experiment

Abstract

Abstract The utilization of accolade awards to motivate reviewers is widespread, but their effectiveness remains uncertain. We aim to explore how receiving an accolade award affects reviewers’ subsequent number of reviews. In contrast to small-scale, unrepresentative experiments, we perform a large-scale, global, and all-disciplinary analysis based on quasi-natural experiments. By integrating the Publons, ORCID, MAG, SciSciNet, and OpenAlex databases, a large dataset that tracks the reviewers’ annual number of reviews and their bibliometric indicators is compiled, encompassing 179,794 individuals. Among them, 6605 individuals who receive the “Publons Global Peer Review Award” in 2018 are constituted as the experimental group. Those non-winners are then matched to form a control group. Using propensity score matching (PSM), 711 and 762 reviewers are matched as the experimental and control groups respectively. An analysis employing the difference-in-differences (DiD) method is conducted to examine the impact of an accolade award on reviewers’ subsequent number of reviews. It is found that, following the receipt of an accolade award, reviewers, on average, reviews about four fewer manuscripts, with the reduction exhibiting a V-shaped pattern. Additional analyses are conducted to examine how individual differences and socio-economic factors influence the awarding effects. Besides, we analyze the mechanisms underlying the awarding effect and propose strategies to motivate reviewers. In brief, the sharp decrease in marginal utility of accolades, the voluntary nature of peer review, and the unexpected properties of accolade awards are the potential mechanisms that generate the negative effect. The academic community should reassess the existing incentive strategies for reviewers.