Evaluating the implementation, mechanisms of effect and context of an integrated care intervention for vulnerable families in Central Sydney Australia: A research framework

Abstract

Introduction: In March 2014, the New South Wales (NSW) Government (Australia) announced the NSW Integrated Care Strategy. The NSW Government’s Integrated Care Strategy funding enabled the establishment of an integrated care initiative called Healthy Homes and Neighbourhoods (HHAN). The Initiative was designed as a population-based, family-centered, care-coordination network that functioned across agencies to assist vulnerable families to navigate the health and social-care system, to keep themselves and their children safe, and in doing so, promote social cohesiveness. The evaluation framework is described. Methods: Since integrated care initiatives are an example of complex interventions in health and social care, we underpinned this programme of development, research and evaluation by the critical realist metatheory of science and the NSW Health Monitoring and Evaluation Framework, using the UK Medical Research Council (MRC) Framework for evaluating complex interventions.1 We adapted the iterative approach of development, feasibility/piloting, evaluation and implementation of the MRC framework to include critical realist, participatory, continuous and theory driven elements. Discussion: Both the 2008 UK Medical Research Council framework for evaluating complex interventions, and our previously reported critical realist methodology, provided the tools to enable an iterative approach to development, design, testing and continuous evaluation. The philosophical paradigms underpinning each approach are different. Both the NSW Health and MRC frameworks were developed from in the “logical positivist” inductive and deductive reasoning traditions with a strong focus on activities and outcomes but little emphasis on process. That weakness was been recognised and the MRC subsequently endorsed advice on process evaluation for complex interventions which accommodated the realist approach incorporated here. Conclusion: Integrated care initiatives are examples of complex interventions in health and social care. The interventions are multi-layered and operationalization is strongly influenced by both historical and current context. This aspect alone makes it imperative that the methodology used is post-positive and takes ontologically stratified context into account. The NSW Health Monitoring and Evaluation Framework did not make provision for assessment of context or mechanism of effect. We describe here a multilevel approach including a continuous improvement approach through constant comparison and triangulation of mixed method findings. Utilising the MRC framework for evaluating complex interventions within a critical realist methodology, enabled us to study both mechanisms of effect and context. Lessons learned: Evaluation of integrated care interventions will require a pluralist approach to methodology. It is possible to design evaluation approaches that allow study of context, mechanism, reach and efficacy in a complex layered design as used for integrated care initiatives. Limitations: The approach utilised was contrained by the requirements of the policy and funding organisation. We did not fully explore the application of Theory of Change and complexity science within the MRC framework. Suggestions for future research: We recommend that further theoretical and applied research be undertaken to advance the methods and tools available for evaluation of complex multi-layered integrated care initiatives. Reference: 1. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655

Keywords

• process evaluation
• theory driven evaluation
• critical realism
• complex intervention