Health and Social Care Delivery Research (May 2024)
A service-user digital intervention to collect real-time safety information on acute, adult mental health wards: the WardSonar mixed-methods study
Abstract
Background Acute inpatient mental health services report high levels of safety incidents. The application of patient safety theory has been sparse, particularly concerning interventions that proactively seek patient perspectives. Objective(s) Develop and evaluate a theoretically based, digital monitoring tool to collect real-time information from patients on acute adult mental health wards about their perceptions of ward safety. Design Theory-informed mixed-methods study. A prototype digital monitoring tool was developed from a co-design approach, implemented in hospital settings, and subjected to qualitative and quantitative evaluation. Setting and methods Phase 1: scoping review of the literature on patient involvement in safety interventions in acute mental health care; evidence scan of digital technology in mental health contexts; qualitative interviews with mental health patients and staff about perspectives on ward safety. This, alongside stakeholder engagement with advisory groups, service users and health professionals, informed the development processes. Most data collection was virtual. Phase 1 resulted in the technical development of a theoretically based digital monitoring tool that collected patient feedback for proactive safety monitoring. Phase 2: implementation of the tool in six adult acute mental health wards across two UK NHS trusts; evaluation via focused ethnography and qualitative interviews. Statistical analysis of WardSonar data and routine ward data involving construction of an hour-by-hour data set per ward, permitting detailed analysis of the use of the WardSonar tool. Participants A total of 8 patients and 13 mental health professionals participated in Phase 1 interviews; 33 staff and 34 patients participated in Phase 2 interviews. Interventions Patients could use a web application (the WardSonar tool) to record real-time perceptions of ward safety. Staff could access aggregated, anonymous data to inform timely interventions. Results Coronavirus disease 2019 restrictions greatly impacted the study. Stakeholder engagement permeated the project. Phase 1 delivered a theory-based, collaboratively designed digital tool for proactive patient safety monitoring. Phase 2 showed that the tool was user friendly and broadly acceptable to patients and staff. The aggregated safety data were infrequently used by staff. Feasibility depended on engaged staff and embedding use of the tool in ward routines. There is strong evidence that an incident leads to increased probability of further incidents within the next 4 hours. This puts a measure on the extent to which social/behavioural contagion persists. There is weak evidence to suggest that an incident leads to a greater use of the WardSonar tool in the following hour, but none to suggest that ward atmosphere predicts future incidents. Therefore, how often patients use the tool seems to send a stronger signal about potential incidents than patients’ real-time reports about ward atmosphere. Limitations Implementation was limited to two NHS trusts. Coronavirus disease 2019 impacted design processes including stakeholder engagement; implementation; and evaluation of the monitoring tool in routine clinical practice. Higher uptake could enhance validity of the results. Conclusions WardSonar has the potential to provide a valuable route for patients to communicate safety concerns. The WardSonar monitoring tool has a strong patient perspective and uses proactive real-time safety monitoring rather than traditional retrospective data review. Future work The WardSonar tool can be refined and tested further in a post Coronavirus disease 2019 context. Study registration This study is registered as ISRCTN14470430. Funding This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: NIHR128070) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 14. See the NIHR Funding and Awards website for further award information. Plain language summary Mental health wards can feel unsafe. We know that patients and staff have different ideas about what makes a hospital ward safe or unsafe. Patients are often the first to know when the atmosphere on a ward becomes tense but, often, no one asks them for input or feedback at the time. We worked with service users and staff to develop new technology to make it easy for patients to tell staff about changes in the ward atmosphere. We put everyone’s ideas together and some technical developers then built a digital safety tool to use on a tablet computer. Patients put in anonymous information about the ward atmosphere and staff can read it straight away. We tested it on six adult acute mental health wards for 10 weeks. We asked patients and staff what they thought about the tool and we looked at how it was being used. Patients and staff liked the look of the tool on the tablet computer. Some staff said they did not need it because they could tell how patients were feeling, but patients told us that staff did not talk with them much and did not always know when patients were feeling tense. Coronavirus disease 2019 made life difficult on the wards. Most ward managers said the tool could be helpful, but they had not had time to get used to it on the wards. Occasionally, the tablet computers were out of action. Many staff tried hard to use the tool. Most patient information was gathered when it was calm, perhaps because staff were not too busy to help them. We found that this tool could help staff know about tensions on the ward, but they need to get used to it and bring it into ward routines. Scientific summary Context Mental health services report high levels of safety incidents. This is a concern and an NHS priority. UK government records for 2020–1 show 300,703 reported incidents in mental health services in England. Incidents in acute mental health wards frequently involve violence and self-harm. Safety incidents have been associated with increased costs and harm to patients and staff. Furthermore, one incident may increase the likelihood of further incidents, via a disturbed ward milieu and social contagion. Successfully avoiding one incident may therefore reduce the probability of future incidents. Patient perspectives on safety highlight factors such as not being listened to, or not feeling psychologically safe; however, incident reporting systems fail to capture the spectrum of patients’ safety concerns. Although patient involvement is a mental health research priority, patient-reported safety data are rarely collected. Onerous processes and fear of repercussions discourage experience experts from participating in mental health research. Patients may experience harm but have difficulties raising concerns with staff. If patients had opportunities to report safety issues in real time, staff could potentially respond and intervene before situations escalate. Interpersonal dynamics on acute mental health wards can change rapidly, affecting care quality and feelings of safety. Potentially, prospective clinical surveillance could promote safety by monitoring rather than simply measuring safety. Proactive day-to-day monitoring of patient perspectives might bring greater benefits than relying on retrospective review, and could be part of a broader vision to improve ward safety. Currently, there is no mechanism by which moment-to-moment safety data from patients on acute mental health wards can be captured and made available to staff in real time. Research aims This study’s focus was to use co-design to develop a digital tool for collecting and monitoring real-time data directly from patients on adult acute mental health wards, and to explore whether this information could be used by staff on a daily basis to anticipate and avoid developing incidents, thereby proactively managing safety. The objectives were: to co-design with service users and staff a digital innovation that will allow real-time monitoring of safety on acute mental health wards; to explore the feasibility and acceptability of capturing real-time feedback from service users about safety; to explore how staff use this information when reported during daily handovers (or other mechanism); to explore how the resulting data are related to quality and safety metrics; to explore how these data can be used longitudinally to promote safety. Methodology Two-phase, mixed-methods design. Literature reviews Scoping review of the literature on patient involvement in safety interventions Systematic searches of academic databases [Cumulative Index to Nursing and Allied Health Literature (CINAHL), PsycINFO, MEDLINE, EMBASE, Web of Science and Scopus] and grey literature (2000–20) were conducted March–June 2020. Additional sources included 14 mental health-specific sources (e.g. Centre for Global Mental Health); 25 non-mental health-specific sources (e.g. Royal College of Nursing); the National Institute for Health and Care Excellence (NICE) evidence database; ProQuest Thesis and Dissertations database and three social media platforms. Following screening, a narrative synthesis of included literature was conducted. A total of 52 studies were included; 33 focused on reducing staff use of restrictive practices. About half reported limited patient participation. Patients were involved mainly as co-thinkers, advisers and partners rather than decision-makers. Safety interventions ranged across organisational change, ward-level decision-making and individual interventions (e.g. mobile phone app). The more extensive their involvement, the more likely patients were to have active roles in the research. High patient involvement seemed to be focused on forensic mental health and associated with reduction in restrictive practices. Low patient involvement tended to be associated with less reduction in restrictive practices. However, methodological quality of the reviewed papers was inconsistent. Evidence scan of the application of digital technology in mental health contexts An evidence scan of the literature around digital technology in a mental health context was conducted in November 2020 (databases: CINAHL, PsycINFO and Web of Science). Research in this field appears largely focused on therapeutic interventions, such as assessment of suicidality, and psychological support therapies, such as counselling. Limited research on telecare and the design of mental health apps for other contexts was identified but digital technologies specifically for mental health care are relatively new. Many digital technologies for health (mainly apps) have been designed within the tech industry, not always using mental health expertise. There was an emphasis on usability and accessibility and there were some concerns around confidentiality. Provided digital technology does not cause patients any harm it is seen as potentially helpful and useful. Successful technology implementation is supported by a reliable internet connection, training for users of the technology, technical support, and effective communication and leadership. Theoretical basis The current study is aligned with patient safety theory, specifically the Measurement and Monitoring Safety Framework domain ‘sensitivity to operations’. This domain, which emphasises monitoring the safety of care as it is delivered in real time, recognises patients and families as important information sources; and highlights the need for staff to be responsive to subtle changes and disturbances. Contagion and milieu were taken as underpinning concepts for promoting ward safety and the development of the safety monitoring tool. A logic model for the WardSonar monitoring tool was developed with a supporting programme theory. Patient and public involvement and engagement A member of the core research team who is a lived experience expert led the approach to patient and public involvement and engagement. This was based around principles of equality, diversity and inclusion. Stakeholder engagement sessions advised on the development of the monitoring tool; critiqued monitoring tool prototypes; and informed the strategies for implementation and evaluation. Study design This was a two-phase mixed-methods design, supported by two NHS trusts. Phase 1: overlapping stages progressed towards conceptual clarification, followed by technical specification and delivery of a testable intervention. Key components were literature reviews, stakeholder interviews and meaningful stakeholder engagement. Technical developers produced a web app that used patient feedback via a tablet computer for proactive safety monitoring: the WardSonar safety monitoring tool. The patient interface employed a weather analogy with questions such as ‘How does the ward atmosphere feel to you today?’ (very calm to very stormy). The staff dashboard displayed aggregated and anonymised information collected via the patient interface in real time. Phase 2: the monitoring tool was implemented in two NHS trusts and evaluated via qualitative interviews, focused ethnography, pre- and post-intervention measures, real-time measures and routinely collected ward data. Two monitoring tools were given to each of the six participating adult acute mental health wards, including two psychiatric intensive care units. Staff were asked to use the tool to invite patients to record their perceptions of the ward atmosphere, three times daily. Changes to protocol The project commenced in the very early stages of the coronavirus disease 2019 (COVID-19) pandemic. Pragmatic adjustments produced a feasible alternative with some virtual Phase 1 data collection and discussions, adjusted timescales and the removal of a phase of small-scale testing. Modes of analysis and interpretation Qualitative evaluation Design Focused ethnographic observations on the six wards explored implementation context. Patient and staff perspectives were explored through individual interviews. Data were synthesised using a pen portrait analytical process. Findings The design of the tool was well received. There was variation between how it was perceived and implemented but general patterns emerged. Patients liked the opportunity to express their concerns and talk with staff who brought the tool to them, explaining that they believed staff were often unaware when patients felt unsafe. Most staff thought the tool prompted useful conversations and could be helpful, although psychiatric intensive care unit staff tended to say it was unnecessary. However, the tool’s functionality for informing safety interventions via aggregated patient data was poorly understood. There was a drop off in engagement with the tool over time. Ward managers discussed the difficulties of motivating busy and exhausted staff to embed the technology into ward routines. Quantitative evaluation Design Staff perceptions of safety culture and ward atmosphere were examined using a pre- and post-implementation design. Baseline measurements including the EssenCES© (Climate Evaluation Schema; Institute of Forensic Psychiatry and Sex Research, Essen, Germany) scale and the Agency for Healthcare Research and Quality Hospital Survey on Patient Safety Culture were taken at baseline and after a 10-week implementation period. Various statistical models were used to explore relationships between wards, WardSonar use and occurrence of incidents over time, including zero-inflated negative binomial models and ordinal logistic regression. The design matrix was consistent across all models. It consisted of fixed effects to capture time-invariant ward-specific effects, time of day variables and lagged values of incidents of WardSonar responses, depending on whether the model was explaining current incidents or current response. A simple linear trend determined any systematic deviation in use or response over time. Results Statistical analysis showed substantial and significant variation in the use of the device across wards, both in terms of the likelihood of any submission at a given time and the number of submissions. There were no statistically significant differences in staff perceptions of ward atmosphere or safety culture pre and post WardSonar implementation. Owing to the sample size, analyses were not conducted at the individual ward level. The volume of patient submissions via the monitoring tool corresponded more closely to incidents than ward atmosphere averages. It was lower over time, at night, at weekends, and when there were peaks in incidents. Submission volume decreased over time, but the probability of a submission reporting a better atmosphere increased slightly. Submission volume was higher during the daytime or before an incident and there was weak evidence of increased volume in the hour after an incident. The type of response for both direction and current atmosphere was not sensitive to whether or not there had been an incident. In terms of direction of atmosphere, there were significant differences across wards, but this was the only significant variable. Evenings led to greater likelihood of a worse atmosphere being reported, given that a submission was made. An incident in the previous four hours was strongly predictive of a further incident. An individual ‘stormy’ response or increased volume of submissions within the previous hour had some predictive value regarding a further incident. Qualitative analysis highlighted some data quality issues; for example, staff were more likely to collect submissions during quiet times and very unlikely to collect submissions during busy times; the tool was not used if the device was lost or otherwise out of action. There were some periods of days or weeks on some wards when no submissions were collected. The technology was pragmatically adapted for use within an NHS context. Some connectivity issues were identified, although data inputting did not seem to be affected. Further understanding of relationships between ward atmosphere and staff stress or absence due to sickness, contagion between incidents in the seclusion room and impact on staff and patients on the main ward and implementation processes would inform future implementation of WardSonar and interpretation of WardSonar data. Discussion The WardSonar patient safety monitoring tool can collect real-time data about patients’ perceptions of safety, to support staff in monitoring and improving the clinical environment. The research environment was affected by the COVID-19 pandemic and was severely atypical. Regarding the quantitative analysis, the lower volume of responses may have severely limited the power of the statistical analyses to identify statistically significant relationships between responses and outcomes. The decline in tool use over time may simply be an artefact of the pandemic or may have a number of causes. Previous studies have examined factors affecting participation and attrition in digital health technology research. The volume of routinely collected data was not affected and it is those data in which significant findings were identified. Strengths and limitations WardSonar’s strong patient perspective arguably gives it particular relevance for addressing patient safety. The co-design approach produced a tool that was apparently well designed, acceptable and easy to use. Stakeholder perspectives fundamentally impacted conceptualisation, development and operationalisation, adding to WardSonar’s validity. The separate components of the mixed-methods design were mutually beneficial, resulting in rounded insights into the study data, including strong evidence to support the idea of behavioural contagion between safety incidents. Some technical issues with connectivity cast doubt on the reliability of the tool. The amended research design was curtailed to adapt to the COVID-19 research context. The tools developed within the study used existing technology that was within budget and suitable for the research aims. Some technology challenges may be attributable to the limitations of this tool for data visualisation and could potentially be addressed in the future via a bespoke dashboard. Data were gathered from interviews with patients, staff and observations on each ward at selected time points, but because interviews were opportunistic, it is unclear how representative they were of staff and patient populations. Only the English language was used. It may have been valuable to include patients in the pre and post evaluation EssenCES (Climate Evaluation Schema) and Agency for Healthcare Research and Quality (AHRQ) assessments, although this could have been burdensome. Furthermore, one of the aims of assessing patient safety culture via the staff-facing AHRQ measure was to explore composite measures over time that linked to factors within the logic model and programme theory around how the staff dashboard may support staff in monitoring and improving the safety of the clinical environment. Equality, diversity and inclusion The current study was predicated on principles of equality, diversity and inclusion. Stakeholder perspectives were integral to design and development. The views of people not interviewed are unknown. Likewise, little information was collected concerning personal demographic characteristics of participants. This was a deliberate decision made with respect to the ethical arguments around collecting personal information, and because demographic information was not considered relevant in the current study. Reliance on the English language and the limited diversity data reduce the relevance of the results. Implications for decision-makers Further focus on the tool’s implementation in clinical practice warrants additional research; for example, implementation in a post COVID-19 environment could improve uptake, enhancing data validity. The WardSonar tool can facilitate measurement of contagion and may identify the likelihood of future incidents. The ability to monitor patient perspectives in real time provides a unique, proactive approach to safety. Avoiding reliance on the English language and collecting diversity data could broaden the relevance of the results in future studies. Future technical iterations could refine the staff dashboard and the model for deploying the tools in the ward environment. An approach whereby patients can input data unprompted may improve accuracy and reduce variation in use, enhancing results validity. Further development would require key decisions relating to ownership of the technology. Study registration This study is registered as ISRCTN14470430. Funding This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: NIHR128070) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 14. See the NIHR Funding and Awards website for further award information.
Keywords
