A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering

Abstract

Background: Systematic literature reviews (SLRs) have become a standard practice as part of software engineering (SE) research, although their quality varies. To build on the reviews, both for future research and industry practice, they need to be of high quality. Aim: To assess the quality of SLRs in SE, we put forward an appraisal instrument for SLRs. Method: A well-established appraisal instrument from research in healthcare was used as a starting point to develop the instrument. It is adapted to SE using guidelines, checklists, and experiences from SE. The first version was reviewed by four external experts on SLRs in SE and updated based on their feedback. To demonstrate its use, the updated version was also used by the authors to assess a sample of six selected systematic literature studies. Results: The outcome of the research is an appraisal instrument for quality assessment of SLRs in SE. The instrument includes 15 items with different options to capture the quality. The instrument also supports consolidating the items into groups, which are then used to assess the overall quality of an SLR. Conclusion: The presented instrument may be helpful support for an appraiser in assessing the quality of SLRs in SE.

Keywords

• Systematic reviews
• quality assessment
• critical appraisal
• AMSTAR 2
• systematic literature review
• tertiary study