A rapid mixed-methods evaluation of remote home monitoring models during the COVID-19 pandemic in England

Abstract

Background Remote home monitoring services were developed and implemented for patients with COVID-19 during the pandemic. Patients monitored blood oxygen saturation and other readings (e.g. temperature) at home and were escalated as necessary. Objective To evaluate effectiveness, costs, implementation, and staff and patient experiences (including disparities and mode) of COVID-19 remote home monitoring services in England during the COVID-19 pandemic (waves 1 and 2). Methods A rapid mixed-methods evaluation, conducted in two phases. Phase 1 (July–August 2020) comprised a rapid systematic review, implementation and economic analysis study (in eight sites). Phase 2 (January–June 2021) comprised a large-scale, multisite, mixed-methods study of effectiveness, costs, implementation and patient/staff experience, using national data sets, surveys (28 sites) and interviews (17 sites). Results Phase 1 Findings from the review and empirical study indicated that these services have been implemented worldwide and vary substantially. Empirical findings highlighted that communication, appropriate information and multiple modes of monitoring facilitated implementation; barriers included unclear referral processes, workforce availability and lack of administrative support. Phase 2 We received surveys from 292 staff (39% response rate) and 1069 patients/carers (18% response rate). We conducted interviews with 58 staff, 62 patients/carers and 5 national leads. Despite national roll-out, enrolment to services was lower than expected (average enrolment across 37 clinical commissioning groups judged to have completed data was 8.7%). There was large variability in implementation of services, influenced by patient (e.g. local population needs), workforce (e.g. workload), organisational (e.g. collaboration) and resource (e.g. software) factors. We found that for every 10% increase in enrolment to the programme, mortality was reduced by 2% (95% confidence interval: 4% reduction to 1% increase), admissions increased by 3% (−1% to 7%), in-hospital mortality fell by 3% (−8% to 3%) and lengths of stay increased by 1.8% (−1.2% to 4.9%). None of these results are statistically significant. We found slightly longer hospital lengths of stay associated with virtual ward services (adjusted incidence rate ratio 1.05, 95% confidence interval 1.01 to 1.09), and no statistically significant impact on subsequent COVID-19 readmissions (adjusted odds ratio 0.95, 95% confidence interval 0.89 to 1.02). Low patient enrolment rates and incomplete data may have affected chances of detecting possible impact. The mean running cost per patient varied for different types of service and mode; and was driven by the number and grade of staff. Staff, patients and carers generally reported positive experiences of services. Services were easy to deliver but staff needed additional training. Staff knowledge/confidence, NHS resources/workload, dynamics between multidisciplinary team members and patients’ engagement with the service (e.g. using the oximeter to record and submit readings) influenced delivery. Patients and carers felt services and human contact received reassured them and were easy to engage with. Engagement was conditional on patient, support, resource and service factors. Many sites designed services to suit the needs of their local population. Despite adaptations, disparities were reported across some patient groups. For example, older adults and patients from ethnic minorities reported more difficulties engaging with the service. Tech-enabled models helped to manage large patient groups but did not completely replace phone calls. Limitations Limitations included data completeness, inability to link data on service use to outcomes at a patient level, low survey response rates and under-representation of some patient groups. Future work Further research should consider the long-term impact and cost-effectiveness of these services and the appropriateness of different models for different groups of patients. Conclusions We were not able to find quantitative evidence that COVID-19 remote home monitoring services have been effective. However, low enrolment rates, incomplete data and varied implementation reduced our chances of detecting any impact that may have existed. While services were viewed positively by staff and patients, barriers to implementation, delivery and engagement should be considered. Study registration This study is registered with the ISRCTN (14962466). Funding This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (RSET: 16/138/17; BRACE: 16/138/31) and NHSEI and will be published in full in Health and Social Care Delivery Research; Vol. 11, No. 13. See the NIHR Journals Library website for further project information. The views expressed in this publication are those of the authors and not necessarily those of the National Institute for Health and Care Research or the Department of Health and Social Care. Plain language summary The problem COVID-19 patients can experience very low oxygen levels, without feeling breathless. Patients may not realise there is a problem until they become extremely unwell, risking being admitted to hospital too late. To address this, COVID-19 remote home monitoring services were developed and later rolled out across England. Patients monitored oxygen levels at home using an ‘oximeter’ (a small device which clips on to your finger) and sent these readings to providers via phone or technology (e.g. an app). Patients could access further care if needed. We did not know whether these services worked, or what people felt about them. We looked at •How services were set up and used in England. •Whether services work (e.g. by reducing deaths and length of hospital stay). •How much they cost. •What patients, carers and staff think about these services (including differences between groups and telephone vs. technology). What we did We looked at available existing evidence and collected data from eight services operating in the first wave of the pandemic. During the second wave of the pandemic, we used data available at a national level and conducted surveys (28 sites) and interviews (17 sites) with staff, patients and individuals involved in developing/leading services nationally. What we found These services have been used worldwide, but they vary considerably. We found many things that help these services to be used (e.g. good communication) but also things that get in the way (e.g. unclear referrals). Our findings did not show that services reduce deaths or time in hospital. But these findings are limited by a lack of data. Staff and patients liked these services, but we found some barriers to delivering and using the service. Some groups found services harder to use (e.g. older patients, those with disabilities and ethnic minorities). Using technology helped with large patient groups, but it did not completely replace phone calls. Conclusion Better information is needed to know whether these services work. Staff and patients liked these services. However, improvements may make them easier to deliver and use (e.g. further staff training and giving additional support to patients who need it). Scientific summary Background and rationale Delays in the presentation of patients with COVID-19 has led to patients arriving at hospital with very low oxygen saturations often without breathlessness (‘silent hypoxia’). This has resulted in patients being admitted to hospital with advanced COVID-19, thus requiring invasive treatment, potential admission to intensive care and poorer outcomes. Remote home monitoring models that systematically record and communicate patients’ physiological parameters to clinicians are currently being used globally for a variety of conditions. These models offer a potential solution for reducing the delays in providing appropriate treatment for patients with COVID-19 by identifying at-risk patients earlier. As a result, services providing remote home monitoring using pulse oximetry for patients with COVID-19 were developed ad hoc in some areas in England during the first wave of the pandemic (March to July 2020). Learning from these earlier services, NHS England and NHS Improvement (in November 2020) launched a national roll-out of a model of care called ‘COVID Oximetry @home (CO@h)’, followed by early discharge models, referred to as ‘virtual wards’ (in January 2021). We refer to these services as COVID-19 remote home monitoring services. All of these services provide patients with an oximeter and ask them to regularly record and relay their oxygen levels (alongside other generic COVID-19 symptoms) to a supporting team of administrators and clinicians via a smartphone application (app), e-mail or online portal, or over the telephone. Patients being monitored are escalated to receive additional care if necessary. Previous research has explored remote home monitoring for other conditions, but there is a lack of research on the effectiveness, cost, implementation and staff/patient experiences of remote home monitoring models for COVID-19. This study explored the impact and implications of these COVID-19 remote home monitoring services during the first and second waves of the pandemic. Phase 1 Phase 1 of this evaluation (during the first wave of the pandemic) aimed to answer the following research questions: 1.How have remote home monitoring services been implemented for COVID-19 and what are their main components, processes of implementation, target patient populations, impact on outcomes, costs and lessons learned? 2.What were the characteristics of remote home monitoring models for COVID-19, experiences of staff implementing these models, data processes and lessons learned during wave 1 of the pandemic? Phase 2 Phase 2 of this evaluation (during wave 2 of the pandemic) aimed to answer the following research questions: 1.Are COVID-19 remote home monitoring services associated with changes in mortality and use of hospital services? Does the use of tech-enabled oximetry have a measurable effect on mortality and hospitalisations? 2.What were the costs of setting up and running COVID-19 remote home monitoring services and how do these costs vary between tech-enabled and analogue, and analogue-only data submission modes? 3.What are the factors influencing delivery and implementation of COVID-19 remote home monitoring services? Do these vary by type of model, geography, mode of remote monitoring approach (tech-enabled vs. analogue)? 4.What are the experiences and behaviours (i.e. engagement with services, use of other services) of patients receiving COVID-19 remote home monitoring services? Do these vary by type of model, patient characteristics, mode of remote monitoring (tech-enabled vs. analogue)? 5.Are there potential impacts on inequalities? 6.What are the experiences of staff delivering COVID-19 remote home monitoring services? Do these vary by mode of remote monitoring (tech-enabled vs. analogue)? Methods This study used mixed methods consisting of two phases. Phase 1 Phase 1 (data collected between July and August 2020) comprised a rapid systematic review (n = 27 articles) and an empirical mixed-methods implementation study of staff experiences, the use of data for monitoring progress against outcomes, variability in staffing and resource allocation, patient numbers and impact and lessons learnt (in eight sites). Phase 2 Phase 2 (data collected between January and June 2021) was a large-scale, multisite, mixed-methods study, including: effectiveness, cost analysis, implementation and patient/staff experience (in 28 sites). To explore impact and effectiveness of remote home monitoring services relating to hospitalisations and mortality, we used routinely available data, hospital administrative data and aggregated and other information produced by the programme. To explore costs of setting up and running COVID-19 remote home monitoring services, we collected aggregated data on patient numbers, staffing models, and allocation of resources from 26/28 sites. To explore implementation, staff experiences of delivering these services, patient experiences of receiving and engaging with these services (including a focus on inequalities and technology-enabled and analogue vs. analogue-only models), we conducted surveys and interviews with staff, patients and carers, and interviewed national leads. We involved patients, carers and the public throughout the project. Members of the study team met with members of the Birmingham, RAND and Cambridge Evaluation Centre and Rapid Service Evaluation Team patient and public involvement groups throughout the project (four meetings), to discuss various aspects of the project, including but not limited to the research questions, data collection tools and findings. Results A summary of findings is provided in Figure A. FIGURE ASummary of key findings. Phase 1 Findings from the systematic review indicated that remote home monitoring services have been implemented internationally for COVID-19. Findings from the review and empirical study highlighted that models of remote home monitoring for COVID-19 varied internationally and within England. Many factors facilitated implementation, including good communication within clinical teams, culturally appropriate information for patients and carers, and the combination of multiple approaches for patient monitoring (app and paper based). Findings from phase 1 were disseminated widely and used to inform decisions in relation to the future roll-out of services and the design of phase 2. Phase 2 We received surveys from 292 staff (39% response rate) and 1069 patients/carers (18% response rate), and conducted interviews with 58 staff, 62 patients/carers and 5 national leads. The rapid development of national remote home monitoring services took place in three phases: local development (during wave 1 of the pandemic), national development and roll-out (between waves 1 and 2 of the pandemic) and local implementation (during wave 2 of the pandemic). Despite national roll-out, enrolment of people to COVID-19 remote home monitoring services was lower than expected and there was large variability in the models of remote home monitoring services that were implemented. This variation was influenced by patient, workforce, organisational and resource factors. The overall enrolment rate to the service across 37 clinical commissioning groups judged to have complete data was 8.7%. We found that for every 10% increase in enrolment to the programme, mortality was reduced by 2% (95% confidence interval [CI] 4% reduction to 1% increase), admissions increased by 3% (95% CI −1% to 7%), in-hospital mortality fell by 3% (95% CI −8% to 3%) and lengths of stay increased by 1.8% (95% CI −1.2% to 4.9%). None of these results are statistically significant. For COVID virtual wards (CVW), we found that the roll-out of virtual ward services for COVID-19 did not reduce rates of readmission (adjusted odds ratio 0.95, 95% CI 0.89 to 1.02) or lengths of stay in hospital. In fact, our analysis indicated longer lengths of stay (adjusted incidence rate ratio 1.05, 95% CI 1.01 to 1.09). The mean running cost per patient monitored under the CO@h services was slightly lower compared with CVW services (£527.5 vs. £599.1). For CO@h and CVW services the mean cost per patient monitored at home was lower in sites using both tech-enabled and analogue modes of data submission compared with the sites using analogue-only modes. The majority of staff involved in running COVID-19 remote home monitoring services were clinical staff. Over 50% of staff (clinical and non-clinical staff combined) were employed at band 5 or below in the CO@h service, whereas in CVW services there were slightly more staff on band 6 or above. Staff generally reported positive experiences of delivery (75% of staff reported a positive impact of their role on job satisfaction); they felt that services were easy to deliver and they valued the support provided. However, findings indicated that staff would have benefited from further training; 41% of service leads and 12% of delivery staff identified further training or support needs. Factors influencing delivery of remote home monitoring services for COVID-19 included: staff knowledge and confidence, NHS resources and capacity on staff workload, multidisciplinary team dynamics, and patient (dis)engagement. Patients and carers reported positive experiences (93% rated the service as good or excellent) and felt that services and human contact received as part of these services reassured them and were easy to engage with. Findings indicated that patients with COVID-19 can engage with remote monitoring services but may require support from staff and family/friends to do so. Engagement was conditional on a range of factors including patient factors, support and resources, and service characteristics. Findings indicate that burden of treatment may be experienced by patients and families with acute conditions. Many sites designed their service to be inclusive of the needs of local populations to ensure broad reach, and many sites adapted their service locally to suit specific patient needs to encourage engagement. Despite these local adaptations to services, disparities were reported across patient groups. Age (p < 0.001) and level of education (p < 0.001) were related to whether patients reported a problem with the service, and health status, ethnicity, gender and level of education were associated with engagement with services, and age (p = 0.005) and ethnicity (p = 0.001) were associated with patient reports of understanding information. Most of the services included in this evaluation offered tech-enabled and analogue data submission options to patients. Older patients (p = 0.005), patients with a lower level of educational attainment (p = 0.011) and ethnic minorities (p = 0.043) were more likely to relay symptoms through phone calls with the service. Staff considered the tech-enabled models better equipped to manage large patient numbers; however, many improvements were suggested to improve functionality of technology systems to better fit clinical and operational needs. For patients and staff, tech-enabled and analogue models were not a substitute for human contact, which was a feature of all models. Staff used phone calls to gain comprehensive knowledge of their patients’ condition and ensure they had care in the most appropriate setting. Limitations One limitation of our evaluation was that data were commonly incomplete or absent and services were not used as extensively as expected; therefore, we were unable to conclusively determine the effectiveness of services. Additional limitations included the inability to link data on service use to outcomes at a patient level, low survey response rates and the under-representation of some patient groups. Conclusions Our evaluation was unable to provide conclusive evidence regarding the effectiveness of COVID-19 remote home monitoring services on hospitalisations, lengths of hospital stay and mortality, due to low rates of enrolment and lack of data. Findings also outline large variability in the models implemented in relation to design and intensity of monitoring, workforce, enrolment levels and criteria. A number of factors influenced implementation including patient, staff, organisational and resource factors. Services were viewed positively by staff and patients alike, but some challenges to delivery and engagement have been identified, so services may not be appropriate for all groups without adaptations. Future remote home monitoring services for COVID-19 and other conditions should ensure that staff are well supported and have capacity to deliver these services, that patients have appropriate support, ability, and understanding to engage with these services. Findings from these studies highlighted the need for quality data to be collected as part of future service implementation in order to enable evaluations of effectiveness in future. Future research is needed in several areas. For example, longitudinal evidence on the effectiveness and cost-effectiveness of COVID-19 remote home monitoring services (requiring high-quality complete linked data sets) is needed. Additionally, research on the appropriateness of different models of remote home monitoring services for different groups of patients, and experiences of staff, patients and carers whose views may not have been captured within this evaluation are needed. Study registration This study is registered with the ISRCTN (14962466). Funding This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (RSET: 16/138/17; BRACE: 16/138/31) and NHSEI and will be published in full in Health and Social Care Delivery Research; Vol. 11, No. 13. See the NIHR Journals Library website for further project information. The views expressed in this publication are those of the authors and not necessarily those of the National Institute for Health and Care Research or the Department of Health and Social Care.