Quality of the reviews submitted by attendees of a workshop on peer review

Abstract

Objective: The objective of the study was to study the methodological quality and error detection of the review by the participants of a peer review workshop. Methods: All participants of the workshop were invited to peer review a randomized controlled trial. The manuscript was E-mailed to them after introducing eight deliberate errors to it. Specific instructions and a deadline were provided. All the reviews were analyzed using review quality instrument (RQI). Furthermore, the rate and the type of errors identified were recorded. Results: Of 25 participants, 16 (64%) returned the reviews. The mean total score on RQI was 4.12 (standard deviation 0.70, 95% confidence interval 3.74–4.50); the items which most reviewers did not discuss where the importance of research question and originality of the paper. The number of errors correctly identified varied from 0 to 6 (median 3), the most common being a wrong conclusion (87.5%), randomization procedure (50%), written informed consent (50%), ethics committee approval (42.8%), and masking (31.2%). Only 5 (31.2%) gave an overall recommendation on whether the manuscript should be accepted or not. Conclusions: Major errors were readily identified by the reviewers; however, the need for training was felt in some areas in which the review quality was modest.

Keywords

• Error detection
• methodological quality
• peer review