Public Health Research (Oct 2023)
Investigating health and social outcomes of the Big Local community empowerment initiative in England: a mixed method evaluation
Abstract
Background Most research on community empowerment provides evidence on engaging communities for health promotion purposes rather than attempts to create empowering conditions. This study addresses this gap. Intervention Big Local started in 2010 with £271M from the National Lottery. Ending in 2026, it gives 150 relatively disadvantaged1 communities in England control over £1M to improve their neighbourhoods. Objective To investigate health and social outcomes, at the population level and among engaged residents, of the community engagement approach adopted in a place-based empowerment initiative. Study design, data sources and outcome variables This study reports on the third wave of a longitudinal mixed-methods evaluation. Work package 1 used a difference-in-differences design to investigate the impact of Big Local on population outcomes in all 150 Big Local areas compared to matched comparator areas using secondary data. The primary outcome was anxiety; secondary outcomes included a population mental health measure and crime in the neighbourhood. Work package 2 assessed active engagement in Big Local using cross-sectional data and nested cohort data from a biannual survey of Big Local partnership members. The primary outcome was mental well-being and the secondary outcome was self-rated health. Work package 3 conducted qualitative research in 14 Big Local neighbourhoods and nationally to understand pathways to impact. Work package 4 undertook a cost-benefit analysis using the life satisfaction approach to value the benefits of Big Local, which used the work package 1 estimate of Big Local impact on life satisfaction. Results At a population level, the impacts on ‘reporting high anxiety’ (–0.8 percentage points, 95% confidence interval –2.4 to 0.7) and secondary outcomes were not statistically significant, except burglary (–0.054 change in z-score, 95% confidence interval –0.100 to –0.009). There was some effect on reduced anxiety after 2017. Areas progressing fastest had a statistically significant reduction in population mental health measure (–0.053 change in z-score, 95% confidence interval –0.103 to –0.002). Mixed results were found among engaged residents, including a significant increase in mental well-being in Big Local residents in the nested cohort in 2018, but not by 2020; this is likely to be COVID-19. More highly educated residents, and males, were more likely to report a significant improvement in mental well-being. Qualitative accounts of positive impacts on mental well-being are often related to improved social connectivity and physical/material environments. Qualitative data revealed increasing capabilities for residents’ collective control. Some negative impacts were reported, with local factors sometimes undermining residents’ ability to exercise collective control. Finally, on the most conservative estimate, the cost–benefit calculations generate a net benefit estimate of £64M. Main limitations COVID-19 impacted fieldwork and interpretation of survey data. There was a short 4-year follow-up (2016/20), no comparators in work package 2 and a lack of power to look at variations across areas. Conclusions Our findings suggest the need for investment to support community organisations to emerge from and work with communities. Residents should lead the prioritisation of issues and design of solutions but not necessarily lead action; rather, agencies should work as equal partners with communities to deliver change. Funding This project was funded by the National Institute for Health and Care Research (NIHR) Public Health Research Programme (16/09/13) and will be published in full in Public Health Research; Vol. 11, No. 9. See the NIHR Journals Library website for further project information. Plain language summary The Communities in Control study is looking at the health impacts of the Big Local community empowerment programme, funded by the National Lottery Community Fund and managed by Local Trust (a national charitable organisation). Residents of 150 English areas have at least £1M and other support to improve the neighbourhoods. There have been three phases of the research. This report shares findings from their third phase, which began in 2018. First, we used data from a national survey and data from national health and welfare services to compare changes in mental health between people living in Big Local areas and those in similar areas that did not have a Big Local partnership. Furthermore, we also used publicly available data on crime in the neighbourhoods. We found weak evidence that Big Local was linked with improved mental health and a reduction in burglaries. Second, we used data from a survey conducted by Local Trust to look at health and social impacts on the most active residents. We found an increase in mental well-being in 2018 but this was not maintained in 2020, probably due to the COVID-19 pandemic. Third, we did interviews and observations in 14 Big Local areas to understand what helps and what does not help residents to improve their neighbourhoods. We found that partnerships need to have legitimacy, the right balance of support, and learning opportunities. Residents suggested that creating social connections and welcoming social spaces, improving how people view the area and tackling poverty contributed to health improvements. Direct involvement in Big Local was both stressful and rewarding. Finally, we did a cost–benefit analysis by putting a monetary value on residents’ increase in life satisfaction due to Big Local and comparing it with the costs of Big Local. We found that the benefits exceed the costs by at least £60M, suggesting that Big Local provides good value for money. Scientific summary Background This research was commissioned in response to a call in 2016 by the National Institute for Health and Care Research (NIHR) Public Health Research Programme (PHRP) for research ‘to address urgent gaps in the evidence on which interventions, using a community engagement approach, are effective in improving health and well-being and reducing health inequalities.’ We identified that the most glaring gaps were in relation to community empowerment initiatives. In theory these have great potential for reducing health inequalities, but research has focused instead on initiatives adopting a much narrower model of engagement of communities in professional-led interventions. We were able to take advantage of the Big Local (BL) programme across England, a rare example of a nationwide experiment in community empowerment. Big Local is the largest community empowerment initiative implemented in England. Launched in 2010 with £271M from the National Lottery Community Fund, and ending in 2026, it involves giving 150 relatively disadvantaged communities in England control over £1M each to enable residents to improve their neighbourhoods. It is overseen by a national organisation, Local Trust, and managed by a BL partnership board in each area, with at least 51% resident members. Our Communities in Control (CiC) study is a long-term evaluation spanning 7 years, with the latest 3 years funded by PHRP. It offers an unparalleled opportunity to learn from the BL experience for future community empowerment initiatives, focusing on the effects on health and well-being and on reducing health inequalities. Research objectives Building on two earlier phases in 2014–15 and 2015–17, the objectives of this third phase of the CiC study were to: investigate longer-term population-level health and social outcomes of BL investigate impacts of BL on health and well-being of engaged residents assess changes in collective control over decisions and actions among BL residents and pathways to changes identified illuminate residents’ perspectives on health and well-being impacts and pathways to these conduct an economic evaluation of BL draw out implications for the future design and evaluation of place-based initiatives that aim to increase collective control, particularly in disadvantaged communities. Methods The study adopted a systems theoretical framework. It is a longitudinal mixed-methods evaluation comprising four work packages. Work package (WP) 1 focused on Objective 1. Secondary data were analysed to assess whether BL had any positive impacts on social and health outcomes for populations in BL neighbourhoods. It employed difference-in-differences (DiD) methods comparing changes in outcomes in BL areas to those in comparator areas between 2011–15 and 2016–19. Comparator areas were matched to BL areas based on observed characteristics such as the Index of Multiple Deprivation, population ethnic and age profiles from the 2011 census and the Office for National Statistics (ONS) area classification. The primary outcome, from the Annual Population Survey, was a binary variable taking the value 1 if individuals reported a score of more than 6 in response to the question ‘Overall, how anxious did you feel yesterday?’, where 0 is ‘not at all anxious’ and 10 is ‘completely anxious’. Secondary outcomes included a composite measure of population mental health, recorded crimes and antisocial behaviour. Work package 2 addressed Objective 2, assessing whether BL had any positive health impacts for actively engaged residents using the 2016, 2018 and 2020 waves of a biannual survey of BL partnership members conducted by Local Trust. The analysis used both the repeated cross-sectional sample provided by the survey and a nested cohort of partnership members who completed all three surveys. The primary outcome was the Short Warwick–Edinburgh Mental Well-being Scale and the secondary outcome self-rated general health status. Unfortunately, we were unable to construct comparators as planned using the 2016, 2018 and 2020 Health Survey for England, because the relevant questions were not in the 2018 survey and the survey was not undertaken in 2020. Work package 3 addressed Objectives 3 and 4. First, qualitative in-depth interviews with national BL stakeholders and informants with expertise in place-based policies aimed to identify changes in the BL programme over time and explore the relationship with national policy. Second, continuing our longitudinal qualitative research, we conducted 171 interviews in 14 areas with people actively involved with BL from November 2018 to January 2021 (overlapping with the pandemic). Interviews covered perceptions of processes enabling or constraining residents’ attempts to improve their area and the impact of BL on health and well-being. In the analysis, we drew on the data from earlier phases of the CiC study. We also conducted observations of BL partnership meetings. Work package 4 addressed Objective 5 by conducting a cost–benefit analysis using the life satisfaction approach to value the benefits of Big Local. This approach compares the impact of an intervention on life satisfaction to the impact of an increase in income on life satisfaction. It used the DiD estimate of the impact of BL on life satisfaction from WP1, estimates of annual household income in BL areas from the ONS, and average household sizes in the BL areas from the 2011 census to put a monetary value on the benefits generated by BL. It then compared the benefits to the costs, that is, the funding BL areas received and the value of the unpaid hours provided by BL partnership members (estimated from responses to the biannual survey of BL partnership members). Results For Objective 1, we found limited evidence for population-level social or health impacts in BL areas versus comparators. The estimated impact on the primary outcome – reporting high anxiety yesterday – was small and not statistically significant (–0.8 percentage points, 95% CI –2.4 to 0.7). Similarly, the estimated impacts on the secondary outcomes were small and not statistically significant, with the exception of burglary (–0.054 change in z-score, 95% CI –0.100 to –0.009). However, there is evidence that BL had some effect on reducing levels of anxiety after 2017. Furthermore, in BL areas that had spent more than 80% of their grant by 2019/20 and thus had made the most progress implementing their plan, we found a statistically significant reduction in the composite measure of population mental health (–0.053 change in z-score, 95% CI –0.103 to –0.002), indicating an improvement in mental health. This improvement in mental health was greater in BL areas that had spent more than 80% of their grant and prioritised social activities, while the reduction in burglaries was also larger when the analysis was limited to these BL areas and greater still in those in this group that had focused on environmental activities. For Objective 2, the impact of BL on engaged residents varied between the cohort and cross-sectional data and by subgroups. In the nested cohort there was a significant increase in mental well-being in engaged BL residents in 2018, but this was no longer statistically significant by 2020. However, this may be explained by the fact that the 2020 survey was delivered in the summer during the COVID-19 pandemic when the mental well-being of the whole country had declined. The bias created by the COVID-19 pandemic potentially affects both the outcomes and all of the explanatory variables, making interpretation of the 2020 survey data problematic. However, across all waves – even in the 2020 COVID-19 data point – residents who perceived that people in the area are willing to help each other and those who agreed that collectively they can influence decisions in the area had significantly higher mental well-being scores. Hours volunteered also had a small positive association with increased mental well-being in both the nested cohort and the repeated cross-sectional analyses, for both men and women and for groups with both high and lower levels of education. There is some evidence of unequal benefits across subgroups. For the cohort, there was a significant improvement in the mental health score in 2018 and 2020 among those with one or more degrees but not for those with ‘no degree’. In both study designs, collective control and willingness to help in the area had positive associations with mental well-being. In the repeat cross-section, residents in the lower education group who agreed that they had collective control had a higher mental well-being score in 2018 and 2020 than those who did not. In the cohort, the positive association with collective control was present for both educational groups. Together, these results suggest that the health benefits of BL participation – especially for residents with a lower level of education – may depend on whether they felt a sense of collective control. The results for gender were similarly mixed across the study designs. In the cohort, there was no significant difference in mental well-being over time for women, but men had a significant increase in the short term only (2018). In the repeat cross-section, more highly educated women had a small significant increase in mental well-being in 2020. Collective control was positively associated with higher mental well-being scores for women and – especially – men. Feeling that people in the area were willing to help was positively associated with mental well-being for women but not for men. For Objective 3, we utilised qualitative data to explore changes in collective control. Qualitative, empirical markers derived in earlier phases of CiC revealed increasing capabilities for collective control – defined as different types of power – over time. Residents in all fieldwork sites reported growing confidence in their individual and collective ability to improve the area (power within); greater understanding of the need for alliances to deliver improvements, and enhanced skills in forging and sustaining external relationships (power with); and in all these areas, meaningful improvements were delivered (power to act). Using longitudinal data, we also started to identify new qualitative markers which signalled an evolution of collective control capabilities and milestones of this evolution through three programme stages: initial implementation, developmental and bedding in. However, there was variation across BL areas. Local dynamics – internal and external to BL partnerships – could undermine residents’ ability to develop the forms of power needed to exercise collective control and also limit the potential for collective control capabilities to develop in the wider community. Without input from support workers, other professionals and local organisations, achievements in some BL areas would have been compromised. On the other hand, these actors sometimes also behaved in ways that undermined the development and exercise of collective control capabilities by residents. For Objective 4, qualitative data illuminated residents’ perceptions of BL’s impact on health and well-being. Frequent references were made to positive impacts on mental well-being at individual and collective levels. Varied pathways were identified, including improvements in social connectivity and cohesion, the physical environment and material living standards. However, there were also accounts of negative impacts on individuals, most often arising from problematic relationship dynamics and/or the burden of responsibility associated with BL partnerships, particularly with leadership roles. Residents experiencing difficulties frequently ‘stepped back’ from the partnership – sometimes temporarily, sometimes permanently. This response reflects the complex relationships many had with BL. Despite sometimes significant negative impacts, residents often also stressed positive benefits from involvement. As a national participant observed, part of the reason why community initiatives may encounter challenges or at times fail is not because people involved are ‘not thoughtful’, but because ‘being resident led is hard’. For Objective 5, the economic analysis suggests that the benefits of BL exceed its costs. The estimated benefits vary across the 150 BL areas due to variation in household income, household size and the number of residents in each area. There are also variations in the costs across BL areas. In total, the estimated annual benefits for all BL areas are £270M, so over the 4-year period from 2016 to 2019 the total benefits are £1080M. The total costs are £121M, so the net benefits are £959M. Sensitivity analyses suggest that BL’s impact on life satisfaction needs to be much lower than our current estimate before the net benefit becomes negative. Using the current impact estimate combined with the most conservative assumptions for the other elements of the cost–benefit calculations generates a net benefit estimate of £64M. Conclusions Big Local has spanned a decade of unprecedented cuts in public sector finances, restrictions on welfare benefits and latterly the COVID-19 pandemic. Despite this difficult context, our findings suggest that capabilities for collective control among engaged BL residents had increased considerably, and they had exercised this control to deliver demonstrable improvements in their neighbourhoods. However, the story is complex. Increases in capabilities for collective control were not always linear, and progress varied across BL partnerships. There is tentative quantitative evidence that BL improved mental health and reduced burglaries, especially in areas making the most progress in implementing plans. There is also evidence of improvement in mental well-being in resident partnership members between 2016 and 2018, though with some indication that these benefits were not equally distributed. The qualitative findings reinforce the possibility of quantifiable benefits, with residents highlighting positive social and mental health impacts through various pathways. However, there is also evidence of negative impacts on health and well-being. Using the most conservative assumptions, BL provides a 30% rate of return on the original National Lottery grant of £196,873,499. Our findings are particularly timely, with growing calls in England for a new ‘community power paradigm’ drawing on the BL model and the anticipated publication of government proposals for ‘levelling up’ policies that may extend current policies, devolving more decision-making and resources down to local people via place-based empowerment initiatives. Our findings have implications for future community empowerment initiatives. (1) Investment is needed to develop and sustain smaller-scale community associations and organisations to emerge from and work with communities. Careful targeting of this investment is crucial if the benefits of BL-type empowerment initiatives are to be distributed equally within and across communities. (2) Residents should be in the lead in defining and prioritising issues and designing solutions; they should be partners in, but not necessarily leaders of, action. (3) Local agencies should prioritise working as equal partners with communities over enabling communities to act for themselves. (4) Care needs to be taken that community-based initiatives are not held accountable to externally determined governance standards that undermine the development of capabilities for collective control. Our findings also have implications for future research evaluating community empowerment initiatives. (1) Evaluations need to be conducted over a long time frame and include a cohort of community members and comparator areas. (2) Secondary data need to be supplemented with primary data providing bespoke quantitative measures tailored to the programme. (3) Evaluations need to be sufficiently powered to detect small but important effects and to allow subgroup analyses that take account of variations in local programmes. (4) Evaluations must integrate an equity lens. (5) Evaluations should include longitudinal qualitative components. Funding This project was funded by the National Institute for Health and Care Research (NIHR) Public Health Research programme (16/09/13) and will be published in full in Public Health Research; Vol. 11, No. 9. See the NIHR Journals Library website for further project information.
Keywords
