Public Health Research (Mar 2025)
Using natural experiments to evaluate population health interventions: a framework for producers and users of evidence
Abstract
Background There has been a substantial increase in the conduct of natural experimental evaluations in the last 10 years. This has been driven by advances in methodology, greater availability of large routinely collected datasets, and a rise in demand for evidence about the impacts of upstream population health interventions. It is important that researchers, practitioners, commissioners, and users of intervention research are aware of the recent developments. This new framework updates and extends existing Medical Research Council guidance for using natural experiments to evaluate population health interventions. Methods The framework was developed with input from three international workshops and an online consultation with researchers, journal editors, funding representatives, and individuals with experience of using and commissioning natural experimental evaluations. The project team comprised researchers with expertise in natural experimental evaluations. The project had a funder-assigned oversight group and an advisory group of independent experts. Results The framework defines key concepts and provides an overview of recent advances in designing and planning evaluations of natural experiments, including the relevance of a systems perspective, mixed methods and stakeholder involvement throughout the process. It provides an overview of the strengths, weaknesses, applicability and limitations of the range of methods now available, identifies issues of infrastructure and data governance, and provides good practice considerations. Limitations The framework does not provide detailed information for the substantial volume of themes and material covered, rather an overview of key issues to help the conduct and use of natural experimental evaluations. Conclusion This updated and extended framework provides an integrated guide to the use of natural experimental methods to evaluate population health interventions. The framework provides a range of tools to support its use and detailed, evidence-informed recommendations for researchers, funders, publishers, and users of evidence. Study registration This methodological project was not registered. Funding This project was jointly funded by the Medical Research Council (MRC) and National Institute for Health and Care Research (NIHR), with project reference MC_PC_21009. The work is published in full in Public Health Research; Vol. 13, No. 3. Plain language summary Governments and other organisations often make changes, for example bringing in new laws, new taxes or changes in the way health care is organised. Changes like these, which are not made by researchers, can be called ‘natural experiments’. As long as some people are affected by a change and some are not, researchers may be able to study the health effects of the changes anyway. We call this ‘natural experimental evaluation’. There is already some guidance on how to conduct this type of research, but methods are advancing constantly and this study needed to update the guidance in a new framework. The researchers formed a writing group to do this, made up of people with skills and experience in doing evaluations of natural experiments. The researchers also held workshops and consulted online with a wider group of experts, including people who use the findings of natural experimental evaluations to help decisions about making public policy. This wider group advised on what should be included in the framework. The writing group was assigned to write the final framework. In this framework, the researchers explain key words and phrases. They also explain why it is important to have a broad definition of a natural experiment. The researchers outline key aspects to bear in mind when designing an evaluation. These include identifying the best opportunities for evaluations, understanding natural experiments within their real-world context, using a variety of research methods, obtaining data, involving stakeholders and various other practical issues. The researchers provide an overview of research methods than can be used, including quantitative, qualitative and economic methods and combinations of these methods. They also provide advice about combining evidence from more than one study. The framework will help people design and use evaluations of natural experiments so they can provide good scientific evidence, but also be as useful as possible for making decisions about how to protect and improve the health of populations. Scientific summary Background Natural experiments, defined as events outside the control of researchers that divide populations into exposed and unexposed groups, are a valuable opportunity to evaluate population health and health system interventions. The conduct of evaluations of these natural experiments has increased substantially since guidance was first published by the UK Medical Research Council in 2012. This increase was due to advances in relevant methods, greater availability of large administrative or routinely collected datasets, and a rise in demand for evaluation of natural experimental interventions delivered at a population level. The original guidance and recent summaries of alternative designs for natural experimental studies have primarily focused on quantitative methods for measuring the size or effect of interventions. Therefore, there is a need for an updated and extended framework that provides additional information on designing and planning natural experimental evaluations, the role of qualitative, mixed methods and economic evaluation along with quantitative methods, and considerations for evidence synthesis and accessing and using routinely collected data. The objective of this framework is to provide an integrated guide for the use of a natural experimental approach to evaluate population health interventions and to: raise awareness among researchers of the range of approaches available for evaluating interventions or other exposures as natural experiments; provide information to help intervention stakeholders, for example in local or central government, decide whether a natural experimental evaluation would be useful, and if so of what kind; and provide information to help journal editors, funders and peer-reviewers to understand the strengths and weaknesses of funding proposals for, and articles reporting, natural experimental evaluations. Methods To develop the framework, the researchers convened a writing group comprising population health researchers from a range of disciplines, including epidemiology, health economics, public health and sociology, and with methodological expertise in statistics, qualitative research and evidence synthesis. Firstly they developed a working draft of the framework, with each member of the group being responsible for drafting one or more chapter. The researchers then used online workshops and an online consultation to collect expert opinions on the content and coverage of the draft. Participants in the workshops (n = 21) and consultation (n = 44) comprised researchers and other relevant stakeholders in Europe, Africa, the Americas and Australasia, including members of research funding boards, journal editors, and representatives of national and local governments. The researchers asked participants in the consultation to review each section of the framework, with the additional feedback being used to further refine the content. Results This framework has a broader scope than the previously published guidance. Feedback from the workshops and online consultation was collated and used to revise the content by the writing group. This input helped guide the use of a broad definition of natural experiment, refine a framework for planning and conducting evaluations of natural experimental studies, and specify in detail the role and content of analytic methods in evaluations. The content of the framework presents information to consider when conducting or using natural experimental evaluations, as set out below. Concepts and definitions The study defines natural experiments as events or processes outside the control of a researcher that divide a population into groups with differing degrees of exposure. A natural experimental evaluation uses an event or process associated with the introduction, delivery or withdrawal of an intervention to evaluate the impact of the intervention. The researchers argue that this broad definition is preferable to narrower definitions based on particular designs or methods, as such lists tend to be arbitrary, or on assignment being ‘as-if randomised’ which can be difficult to define or apply in practice. As methods originating from a range of disciplines are used in the evaluation of natural experiments, a glossary is provided to define key terms. Design and planning An adaptation of the MRC/NIHR framework, for the development and evaluation of complex interventions, provides a structure for planning and conducting a natural experimental evaluation and highlights the value of applying a complex systems perspective for the evaluation. Important stages when scoping and planning a natural experimental evaluation are: Identifying and theorising the natural experiment: opportunities include difference in time or place in the presence or level of exposure between otherwise similar subpopulations, policy eligibility criteria that identify some units within a population but not others as exposed, phased implementation of a policy, randomisation used to assign a policy, and flaws in policy implementation. Assessing the evaluability of the natural experiment: using structured engagement with stakeholders to agree on a conceptual model of how the intervention is expected to achieve its impacts, access relevant data, and consider the costs and usefulness of the evidence. The assessment helps identify key uncertainties and limitations, increases the likelihood of obtaining intended results, and ensures that questions addressed are relevant for decision-making. Conducting feasibility studies for the evaluation: assessing whether the evaluation is viable, and the practicalities of implementing the evaluation design, such as whether routinely collected data adequately captures the necessary exposures. Natural experiments typically occur within complex systems that influence health. When evaluating a natural experiment, considering the implications of the context in which the natural experiment exists helps to understand why the intervention succeeds or fails to achieve the intended impact. An evaluation design with a combination of methods, both qualitative and quantitative, is often needed to provide a comprehensive understanding of a natural experiment, with theory and planning required for how to bring different study designs, types of data and analyses can be brought together. As with most evaluation designs, it is good practice to develop a protocol, or methods-appropriate advance study plan, and to place it in the public domain before analysis commences. In evaluations of natural experiments there will be a diverse range of stakeholders involved at different stages of both the natural experiment intervention and the evaluation. Involvement of relevant stakeholders maximises the likelihood of findings being understood, taken up and used for decision-making. Quantitative methods The quantitative methods used will be defined by the research question being investigated and the design features of the evaluation, with a complex systems perspective likely to require a combination of qualitative methods alongside the quantitative methods. A variety of quantitative methods will often be required to address threats to internal validity of the non-randomised natural experiment. An overview of key quantitative methods is provided, avoiding a hierarchy as the choice of methods should be determined by the research questions and the availability of data, with each method having strengths and weaknesses and therefore appropriateness of use determined by the circumstances of the evaluation. Economic evaluation As there are usually resource constraints for implementations of policies, economic evaluations are valuable in conjunction with evaluations of effectiveness of the natural experiment. Designing, conducting and reporting economic evaluations generates specific challenges, including measuring and identifying costs and outcomes, selecting appropriate analytical methods, identifying the time horizon and considerations of equity. Qualitative methods Qualitative methods make an essential contribution to most natural experimental evaluations, with evaluations benefitting from a mixed-method design incorporating qualitative methods throughout. Key components of an evaluation in which qualitative methods should be integrated include describing the intervention, the system or context, developing a theory of change, informing the selection of populations and controls, characterising and selecting outcomes and indicators, generating data on outcomes, understanding mechanisms and mediators, explaining change and understanding stakeholders’ perspectives. The evaluation should be planned to ensure that the qualitative components are incorporated into the evaluation at the appropriate stages of the evaluation to achieve maximum use of the qualitative data gathered. Reporting, critical appraisal and evidence synthesis of natural experimental evaluations Clear reporting of the natural experiment and the evaluation is crucial for the best use and understanding of these studies. Critical appraisal may be required to understand the rigour of an individual study or undertaken as part of a synthesis of evidence. There is no single tool that can fully assess the risk of bias of all natural experimental evaluation study designs. Therefore, the tools available should be considered in terms of their strengths and limitations for the purpose of a given review. Synthesising evidence from natural experimental evaluations requires consideration of how to manage the probable diversity in study design and characteristics. For some review topics it may be more valuable to examine effectiveness in a broader sense rather than an estimated effect size. Data infrastructure and information governance Natural experimental evaluations often use data that were originally collected for other purposes. Negotiating access to such datasets can be a time-consuming, costly and uncertain process, especially if the research involves the linkage of data from multiple sources. A potential solution to this is to establish secure research platforms, trusted research environments, designed to maintain security and confidentiality of the data and provide efficient access to researchers. Recommendations Good practice considerations Good practice considerations are provided for different users of the framework, derived from the content of the updated framework. In condensed form, these recommendations are provided below. All producers and users of natural experimental evaluations should: Understand the design and planning processes of an evaluation of a natural experiment, including how to identify opportunities for natural experimental evaluation, select the most appropriate evaluation approach and assess the feasibility of the evaluation. Consider the variety and importance of stakeholders. Recognise the respective strengths of quantitative, qualitative and integrated analytical approaches, incorporating perspectives from diverse disciplines, such as economics, social sciences, epidemiology, for investigating the impacts of natural experiments. Researchers conducting natural experimental evaluations should: Be aware of the circumstances that are likely to give rise to good opportunities for a natural experimental approach. Adopt methods that are appropriate to the data available and to the processes that determine exposure to the intervention of interest. Consider adopting a systems thinking approach to evaluating natural experiments. Consider using a combination of methods, including alternative methods of effect estimation, robustness checking and a mixture of qualitative and quantitative methods. Adopt open science practices, publishing a protocol or plan of the study in advance in open access journals or repositories. Clearly report the natural experiment event and all stages of the evaluation, including its planning, protocol, analyses and results, using established reporting standards where available ensuring key details are in plain language appropriate for the evidence users. Include a health equity perspective in the evaluation. Be aware that evaluation of the strength of evidence from natural experimental evaluations should be based on detailed appraisal of the strengths and limitations of the study methods, not on broad study labels. Research funders and commissioners supporting and investing in natural experimental evaluations should: Encourage best practice when commissioning or funding natural experimental evaluations, requiring that a protocol or methods appropriate advance study plan is available prior to analysis commencing, findings are published in open access journals and the relevant reporting guidelines are followed. Establish processes within funding bodies to facilitate flexible and timely responses to prospective natural experimental evaluation opportunities. Support capacity building for natural experiments through investment in infrastructure and the workforce. Negotiate with data owners to make routinely collected data both available and linkable to other datasets for evaluations of policies and programmes. When commissioning natural experimental evaluations, be prepared to be flexible and pragmatic and accept that both the evaluability of the natural experiment and the feasibility of the evaluation require assessment, which may result in the full evaluation not being viable. Flexibility may also be required when considering the start date and timescale of the research, as policy interventions can be delayed, changed or withdrawn, the effects of each will require consideration in an evaluation. Journal editors, policy-makers, practitioners and other decision-makers publishing and using evidence from natural experimental evaluations should: Provide guidance for authors and reviewers on requirements for reports of natural experimental evaluations. Use evidence from high-quality natural experimental evaluations when this is the most appropriate or available form of evidence, being aware of any limitations of the evaluation. Incorporate evaluation plans into the implementation of new policies and programmes. Conclusion The new framework has been developed in consultation with international experts in natural experimental evaluations. It aims to promote the conduct and application of methodologically robust studies where a policy or programme is amenable to evaluation as it is or has been implemented. The researchers hope it will raise awareness of the whole range of issues that need to be considered when planning an evaluation or using the results to influence policy. With the large number of topics covered by the framework, the study aimed to convey the key points with signposting to more detailed information provided throughout. Funding This project was jointly funded by the Medical Research Council (MRC) and National Institute for Health and Care Research (NIHR), with project reference MC_PC_21009. The work is published in full in Public Health Research; Vol. 13, No. 3.
Keywords
