Whole-school interventions promoting student commitment to school to prevent substance use and violence, and improve educational attainment: a systematic review

Abstract

Background Whole-school interventions modify the school environment to promote health. A subset of these interventions promotes student commitment to school to prevent substance (tobacco, alcohol, other drugs) use and/or violence. A previous review identified the theory of human functioning and school organisation as a comprehensive theory of such interventions, and found evidence that these interventions reduce substance use and/or violence. Objectives The objectives were to search for, appraise and synthesise evidence to address the following questions: (1) What whole-school interventions promoting student commitment to school to prevent substance use and/or violence have been evaluated, what intervention subtypes are apparent and how closely do these align with the theory of human functioning and school organisation? (2) What factors relating to setting, population and intervention affect implementation? (3) What are the effects on student substance use, violence and educational attainment? (4) What is the cost-effectiveness of such interventions? (5) Are intervention effects mediated by student commitment to school or moderated by setting or population? Data sources A total of 56 information sources were searched (in January 2020), then an updated search of 48 of these was carried out (in May 2021). Reference lists were also searched and experts were contacted. Review methods Eligible studies were process/outcome evaluations of whole-school interventions to reduce student violence or substance use among students aged 5–18 years attending schools, via actions aligning with the theory of human functioning and school organisation: modifying teaching to increase engagement, enhancing student–staff relationships, revising school policies, encouraging volunteering or increasing parental involvement. Data extraction and quality assessments used existing tools. Theory and process reports were synthesised qualitatively. Outcome and economic data were synthesised narratively; outcome data were meta-analysed. Results Searches retrieved 63 eligible reports on 27 studies of 22 interventions. We identified four intervention subtypes focused on student participation in school-wide decisions, improving staff–student relationships, increasing engagement in learning and involving parents. The theories of change of most intervention subtypes aligned closely with the theory of human functioning and school organisation, and informed refinement of an intervention theory of change. Theories of change for interventions increasing learning engagement did not align with this theory, aiming instead to increase school commitment primarily via social skills curricula. Factors influencing the implementation included whether or not interventions were tailorable, workable and well explained. Interventions with action groups comprising staff/students, etc. and providing local data were well implemented. Implementation was also affected by whether or not schools accepted the need for change and staff had the resources for delivery. Meta-analyses suggest small, but significant, intervention effects in preventing violence victimisation and perpetration, and substance use. There was sparse and inconsistent evidence of moderation and some evidence of mediation by student commitment to school. Two economic evaluations suggested that there is the potential for the interventions to be cost-effective. Limitations The quality of the studies was variable and the economic synthesis was limited to two studies. Conclusions Whole-school interventions aiming to promote student commitment to school share similar theories of change and factors affecting implementation. They have the potential to contribute to preventing violence and substance use among young people. Future trials should aim to optimise intervention effectiveness by better theorisation, and assess implementation and effect moderators and mediators. Study registration This study is registered as PROSPERO CRD42019154334. Funding This award was funded by the National Institute for Health and Care Research (NIHR) Public Health Research programme (NIHR award ref: 17/151/05) and is published in full in Public Health Research; Vol. 12, No. 2. See the NIHR Funding and Awards website for further award information. Plain language summary Whole-school health interventions aim to modify how schools are run, to promote students’ health. Some aim to promote student commitment to school to prevent the important interlinked outcomes of substance (tobacco, alcohol, other drugs) use and violence. We searched for all evaluations of such interventions. We summarised what this research said about the sorts of interventions used, how they are meant to work, what factors affect delivery, whether or not they reduce violence and substance use and whether or not they are worth the money. We found 63 reports on 27 studies of 22 interventions. We identified four subtypes of interventions. These aimed to involve students in school decisions, improve staff–student relationships, increase engagement in learning or involve parents. Most of these interventions were intended to work by making sure schools focused on student needs, or by improving relationships between staff and students, between different areas of learning or between schools and communities. This aimed to make students feel committed to school and therefore avoid violence or substance use. A few aimed to work mostly by teaching students how to avoid violence and substance use. We found that interventions were well implemented if they were tailored for each school and had good materials and support. Interventions were well delivered if they were led by action groups (comprising staff, students, etc.) or provided schools with information on students’ needs. Implementation was affected by whether or not schools accepted the need for change and whether or not staff had the necessary time and money to do the work. These interventions appear to have small, but significant, intervention impacts in preventing violence and substance use among young people. There was not consistent evidence of different effects for different students. A small number of studies suggest that such interventions might show economic benefit, but this would need further research. Future research should focus on interventions that are refined to make sure that they can be well delivered. Scientific summary Background and rationale Substance use and violence are important interconnected outcomes often associated with disengagement from school. Whole-school interventions aiming to modify the school environment to promote student commitment to school might be an effective way to prevent these outcomes. This review synthesised evidence on such interventions. According to a previous review, the theory of human functioning and school organisation provides the most comprehensive theory of how schools can influence student commitment to school and health behaviours. This proposes that promoting student commitment to school can decrease involvement in risk behaviours by promoting student practical reasoning and positive peer affiliations. Schools can promote commitment by reframing provision on student needs and eroding boundaries between staff and students, between different areas of learning and between schools and local communities. We used this theory to define our initial theory of change and inclusion criteria, and as a starting point for categorising interventions and synthesising theories of change. Aim and review questions The aim was to search systematically for, appraise and synthesise evidence to address the following questions: What whole-school interventions promoting student commitment to school to prevent student substance use and/or violence have been evaluated, what subtypes are apparent and how closely do these align with the theory of human functioning and school organisation? What factors influence implementation of such interventions? Overall and by intervention subtype, what are the effects of such interventions on student substance use, violence and educational attainment? What is the cost-effectiveness of such interventions? Are the effects of such interventions on student substance use and/or violence mediated by student commitment to school, or moderated by setting or population? Methods Inclusion criteria Eligible studies were process or experimental/quasi-experimental outcome evaluations of whole-school interventions aiming to reduce violence or substance (tobacco, alcohol, other drug) use among students aged 5–18 years in schools, by modifying teaching to increase engagement in learning, enhancing student–staff relationships, revising school policies, encouraging student volunteering or involving parents in school. Searching information sources The search strategy included terms covering population, intervention and evaluation design. We initially searched 21 databases, 3 trial registries and 32 websites (16–27 January 2020), and updated the search across 14 databases, 2 trial registries and 32 websites (11–25 May 2021). We searched reference lists and contacted subject experts. Information management and study selection Citations identified by our searches were de-duplicated before being uploaded to EPPI-Reviewer 4.0 software (Evidence for Policy and Practice Information and Co-ordinating Centre, University of London, London, UK). Two reviewers double-screened batches of 50 references. Disagreements were resolved by discussion. After reaching an agreement rate of ≥ 90% on these batches, each subsequent reference was single-screened on title/abstract. If references were considered likely to be eligible based on the title/abstract, the full report was reviewed using a similar process. Data extraction Two reviewers independently extracted data using existing tools. Disagreements were resolved by discussion. For intervention descriptions, date were extracted on domains included in a standard framework. For theory reports, we extracted data on constructs, mechanisms and contextual contingencies affecting these. For empirical studies, we extracted data on basic study details, methods, interventions and findings. Assessments of quality and risk of bias The quality of each report was assessed independently by two reviewers using existing tools. The reviewers met to compare assessments and resolve any differences through discussion. Theory reports were assessed on the basis of whether or not this described the path from intervention to outcomes, clarity of constructs, clarity of inter-relationships between constructs, whether or not the underlying mechanisms were explained, and whether or not the theory considered how mechanisms and outcomes might vary by context. Process evaluations were assessed on the basis of rigour of sampling, data collection and data analysis; the extent to which findings were grounded in data; whether or not the study privileged the perspectives of students; and breadth/depth of findings. Outcome studies were assessed for risk of bias on the basis of sequence generation, allocation concealment, blinding of participants or personnel, blinding of outcome assessors, incomplete outcome data, selective outcome reporting and other sources of bias. Economic evaluations were assessed using an adapted version of an existing tool. Data analysis To create a categorisation of intervention subtypes, we drew on descriptions of interventions and theories of change, using these to refine an initial categorisation informed by the theory of human functioning and school organisation. Reviewers grouped interventions into discrete categories and subcategories. To synthesise theories of change, we used best-fit framework synthesis. We defined a priori themes based on the theory of human functioning and school organisation and then coded data from included studies against these. When concepts from the included studies could not be coded with these, we coded inductively. This coding was then used to refine the existing theory of change by intervention subtype. We synthesised qualitative and quantitative elements of process evaluation reports using thematic synthesis methods. We conducted a narrative synthesis of outcome evaluations ordered by outcome then, within this, by intervention subtype, follow-up time and study design. Outcomes were categorised into violence, use of tobacco, alcohol and other drugs, and academic attainment. We produced forest plots for different outcomes and follow-up times. Plots included point estimates and standard errors for each study, expressed as standardised mean differences (Cohen’s d) to ensure comparability across reports. When data allowed, we calculated pooled effect sizes within each pairwise comparison, accounting for the extent of heterogeneity among the studies, estimating separate models for each outcome. We regarded follow-up times of up to 1 year and > 1 year post baseline as different outcomes, pooling first by follow-up times and, when appropriate, overall across follow-up times. We used the Grading of Recommendations Assessment, Development and Evaluation to present the quality of evidence. For synthesising economic evaluations, measures of costs, indirect resource use and cost-effectiveness were summarised in a table and adjusted for currency and inflation to the then-current UK context. These data were used to inform a narrative synthesis of economic evidence. Stakeholder consultation We consulted policy and practice stakeholders once during the review. In October 2021, stakeholders reviewed slides summarising the main findings. We asked stakeholders to advise on interpretation, implications and knowledge exchange. Ethics approval The research involved no human participants and drew solely on evidence in the public realm, so ethics approval was not required. Results Included studies The original searches retrieved 62,742 unique references and 56 eligible reports. The updated search retrieved 9709 unique references and 9 eligible reports. In total, 63 reports on 27 studies of 22 interventions were included: 63 on theories of change, 16 on process evaluations, 48 on outcome evaluations and 3 on economic evaluations. Of the included interventions, 10 addressed violence (with 2 of these also addressing educational attainment), 3 addressed substance use (with 1 of these also addressing educational attainment) and 9 addressed violence and substance use (with 1 of these also addressing educational attainment). What interventions have been evaluated and what intervention subtypes are apparent? We identified four intervention subtypes, focused on the following: developing student participation in school-wide decisions, improving staff–student relationships, increasing student engagement in learning and increasing parental involvement in school. How closely do intervention theories of change align with the theory of human functioning and school organisation? We synthesised interventions developing student participation, enhancing staff–student relationships and involving parents together because of commonalities in theories of change. Despite only one intervention being explicitly informed by the theory of human functioning and school organisation, theories of change aligned closely with this theory, with recurrent themes that reciprocally translated with this theory and each other. Inductive coding suggested a number of refinements to the starting theory, for example relating to how school organisation, learning, discipline and school–community links were reframed. Interventions aimed not only to reduce risk behaviours but to promote positive overall development via increasing commitment to school. The theories of change for interventions developing student engagement in learning aligned much less with the theory of human functioning and school organisation. These interventions were theorised to increase students’ school engagement primarily through social skills curricula, with whole-school elements secondary. What factors relating to setting, population and intervention influence the implementation of these interventions? Evaluations suggested that school staff were more likely to understand what was required to implement an intervention when provided with good materials and support. Whether or not staff would commit to delivery was influenced by whether interventions could be integrated with existing practices, locally tailored or built on existing work. Interventions providing local data helped build buy-in. Staff were more likely to commit when external providers were credible and experienced. Students were more likely to commit if an intervention offered opportunities for active roles or expressing their views. School leaders were more likely to commit to an intervention that addressed an issue they were already interested in tackling and when there was already a recognition of the need for change. To ensure those in schools worked together to deliver interventions, interventions needed to be locally workable, fitting with timetables and providing clear guidance. Interventions that included ‘action groups’ (consisting of staff, students, etc.) helped build collective action. Interventions with synergistic components were implemented more successfully. Whether or not staff could come together to deliver interventions also depended on having the time, budgets, authority and connections to support this. Action groups enabled members to reflexively monitor implementation and assess what else was needed. This could give participants the permission and resources to try different things, persisting with what worked. What are the effects on student substance use, violence and educational attainment of whole-school interventions aiming to promote student commitment? Overall, meta-analyses suggested that interventions led to a small, but statistically significant, reduction in violence perpetration at up to 1 year post baseline [odds ratio (OR) 0.85, 95% confidence interval (CI) 0.76 to 0.96] and > 1 year post baseline (OR 0.79, 95% CI 0.65 to 0.98). Considering intervention subtypes, meta-analyses suggested possible, but inconsistent, impacts on violence perpetration of interventions promoting student participation in school policy decisions and student relationships with teachers at up to 1 year post baseline and at > 1 year post baseline. Meta-analyses suggested that interventions promoting student engagement in learning and parental involvement were unlikely to affect violence perpetration up to 1 year post baseline and at > 1 year post baseline. Overall, meta-analyses suggested that interventions led to small, but statistically significant, reductions in violence victimisation at up to 1 year post baseline (OR 0.84, 95% CI 0.72 to 0.98) and > 1 year post baseline (OR 0.85, 95% CI 0.73 to 0.99). Considering subtypes, evidence suggested possible but inconsistent impacts of interventions promoting student participation in decisions and student relationships with teachers on violence victimisation at up to 1 year post baseline and > 1 year post baseline. Studies suggested that interventions promoting parental involvement were unlikely to affect violence victimisation up to 1 year post baseline and > 1 year post baseline. Studies of intervention effects on observed violence were not meta-analysed owing to heterogeneity of study designs, but evidence suggested that interventions promoting student participation in decisions significantly reduced observed violence at > 1 year post baseline. Interventions promoting parental involvement significantly reduced observed violence at up to 1 year post baseline and at > 1 year post baseline. Overall, meta-analyses suggested that interventions led to small, but statistically significant, reductions in substance use at up to 1 year post baseline (OR 0.83, 95% CI 0.70 to 0.97) and at > 1 year post baseline (OR 0.79, 95% CI 0.62 to 0.998). Meta-analyses of such interventions suggested small and statistically non-significant reductions in measures of specific outcomes (alcohol, smoking, illicit drugs, general substance use) at up to 1 year post baseline and at > 1 year post baseline. Considering subtypes, the evidence suggested possible, but inconsistent, impacts of interventions promoting student participation in decisions in reducing substance use at up to 1 year post baseline and at > 1 year post baseline, with similar findings for alcohol outcomes and illicit drug use outcomes separately. There was less evidence of impact on tobacco outcomes. An analysis of omnibus substance use outcomes suggested a larger, but still substantially heterogeneous and non-significant, effect. One trial reported on the effects of an intervention promoting student–teacher relationships: up to 1 year post baseline, there were no significant effects on alcohol or cannabis outcomes, and no significant effects on any reported lifetime smoking, but there was a significant effect on regular smoking; at > 1 year post baseline, there were no significant effects on alcohol use, tobacco smoking or cannabis use outcomes. Findings from two randomised trials suggested possible, but inconsistent, impacts of interventions promoting student engagement in learning in reducing substance use, but a non-significant impact on reducing alcohol use at up to 1 year post baseline. Evidence suggested possible, but inconsistent, impacts of interventions promoting student engagement in learning in reducing substance use at > 1 year post baseline, and a non-significant impact of such interventions on reducing alcohol use at > 1 year post baseline. Our syntheses of effects on academic attainment are exploratory. All such studies were of interventions promoting student participation in decisions and reporting attainment at > 1 year post baseline. Findings did not suggest clear evidence of impact on academic attainment. Are the effects of whole-school interventions on student substance use and violence mediated by student commitment to school, or moderated by setting or population? Evidence was sparse and inconsistent across trials for moderation of effects on all outcomes by population and school characteristics. Three studies examined mediation of intervention effects by student commitment to school. All focused on interventions promoting student participation in decisions. There was some evidence that student commitment to school mediated intervention effects on violence and substance use outcomes. What is the cost-effectiveness of such interventions, overall and by intervention subtype? Economic evidence was sparse but suggested that two interventions aiming to involve students in decision-making were cost-effective. Conclusions Whole-school interventions aiming to promote student commitment to school to prevent violence and substance use can be categorised into those promoting (1) student participation in decision-making, (2) staff–student relationships, (3) engagement in learning and (4) parental involvement in school. Despite not generally being explicitly informed by the theory of human functioning and school organisation, most interventions are underpinned by theories of change closely aligning with this. Factors influencing implementation include whether or not interventions were tailorable, workable, well explained and had synergistic components. Implementation was also affected by whether or not schools accepted the need for school transformation and whether or not staff had the necessary time, budgets, authority and connections for delivery. Interventions were effective in preventing violence victimisation and perpetration and substance use, although effects are small. Such interventions may contribute to broader efforts to reduce violence and substance use among young people but are unlikely to achieve large population impacts alone. Future trials should aim to optimise intervention effectiveness by better theorisation, and assess implementation and effect moderators and mediators. Study registration This study is registered as PROSPERO CRD42019154334. Funding This award was funded by the National Institute for Health and Care Research (NIHR) Public Health Research programme (NIHR award ref: 17/151/05) and is published in full in Public Health Research; Vol. 12, No. 2. See the NIHR Funding and Awards website for further award information.