Health and Social Care Delivery Research (Jul 2023)
Rapid evaluation of service innovations in health and social care: key considerations
Abstract
This essay explores the emerging field of rapid evaluation in health and social care, drawing out the ways in which it is distinctive from or similar to other forms of applied research. It draws closely on the experience of two National Institute for Health and Care Research rapid service evaluation teams established in 2018, of which all authors of this essay are members. Reflection on the rapid evaluation teams’ body of work is used to highlight methodological and operational considerations for researchers, practitioners and policy-makers interested in the commissioning, conduct and application of rapid evaluations in health and social care. The essay is organised according to the ‘evaluation pathway’, from project selection and scoping, stakeholder involvement, research methods and activity, through skills and team working to feedback and dissemination. Its conceptual framing draws on Riley et al.’s rapidity, responsiveness and relevance framework, proposing a fourth dimension of rigour.1 Novel insights include: the primacy of scoping thoroughly a rapid evaluation; the trade-offs to be made in respect of patient and community engagement and rapid working; the opportunities presented by virtual and mixed research methods; the particular challenges of supervising and leading multiple studies undertaken simultaneously; and the need for flexible and dynamic approaches to disseminating emerging findings in as close to real time. The essay concludes by setting out key lessons about how to conduct rapid evaluations. Funding This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (RSET: 16/138/17; BRACE: 16/138/31). Scientific summary Rapid evaluation is increasingly used in health and social care to provide evidence about the design, implementation, experience and early outcomes of service innovations in a complex and typically fast-moving context. The methods used in rapid evaluation are often those of applied health and care research, adapted to enable or mitigate the rapid approach. This essay reports on the work over five years of two National Institute for Health and Care Research (NIHR)-funded rapid service evaluation teams [Birmingham RAND and Cambridge Rapid Evaluation (BRACE) team and Rapid Service Evaluation Team (RSET) from University College London and the Nuffield Trust)] using the combined experience of undertaking 19 studies to assert the importance of responsiveness, relevance and rigour in the practice of rapid evaluation (the ‘4 Rs’, drawing on the work of Riley et al.).1 Rapid evaluation is a field of research that is subject to debate in relation to its definition, design and implementation. Within the overall conceptualisation of the 4 Rs, ‘rapid evaluation’ can be defined as: early evaluation of an intervention, an evaluation with a short timescale, a study that is mobilised more quickly or one that provides early and rapid cycles of feedback. In addition to this timescale-based typology, the experience of the two NIHR-funded teams is that rapid evaluation can helpfully be conceptualised within a relational-based approach where it may be the precursor to a longer study, the follow-up to a longer study, a nested project within a longer-term evaluation or a stand-alone research project. A methodology-based typology may also be applied when planning, designing and undertaking rapid evaluation. Hence researchers may: use a methodology designed specifically for rapid evaluation, increase rapidity of the evaluation by doing less, use alternative technologies and/or data to increase the speed of an existing evaluation method, or adapt methods from a non-rapid evaluation. These typologies are not exhaustive but are offered as an illustration of the wider and sometimes contested debate about different ways to define, plan and undertake rapid evaluation research. An analysis is made of the methodological challenges specific to rapid evaluation with a particular focus on the selection, scoping and co-design of such studies. It is asserted that the process of scoping evaluations must be thorough and inclusive, being critical to determining when a rapid approach is required and if so, the questions to be examined and methods to adopt. Despite this process of scoping in close partnership with stakeholders, trade-offs will be inevitable in rapid evaluation studies and may lead to more uncertain or ambiguous findings. Researchers must therefore be clear and open about the limitations of their findings and ensure a clear focus on using a flexible and proportionate approach. A larger multi-skilled and experienced team will be required for rapid evaluation and especially where multiple projects are being undertaken in parallel. This in turn calls for senior academic and robust project management expertise and capacity to support effective team-working, complex and sometimes politicised stakeholder engagement, and the assurance of high-quality research delivery. The skills required of these teams extend beyond those usually taught in health and care research development programmes and encompass rapid uses of qualitative, quantitative and mixed methods, and proficiency in interpersonal skills for effective engagement, communication and dissemination. The importance of ensuring rapidity across all stages of the evaluation approach is emphasised, including: how to speed up data collection and analysis while preserving rigour; planning dissemination, uptake and impact from the earliest stages of scoping; and working in close partnership with patient, public and professional stakeholders to co-design the aims, approach and outputs. Outcomes of rapid evaluation may include assertion that intended benefits are likely to accrue, proposals for modifying or even halting a service innovation, planning a longer term study, or offering advice to local stakeholders about next steps. In all cases, the need for rapidity, responsiveness and relevance identified by Riley et al. will underpin the approach along with the additional dimension of rigour devised from the experience of the BRACE and RSET teams.1 The essay concludes with methodological and operational lessons for the conduction of rapid evaluation. Funding This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (RSET: 16/138/17; BRACE: 16/138/31).
Keywords
