Health Technology Assessment (Apr 2024)
Evaluation of venous thromboembolism risk assessment models for hospital inpatients: the VTEAM evidence synthesis
Abstract
Background Pharmacological prophylaxis during hospital admission can reduce the risk of acquired blood clots (venous thromboembolism) but may cause complications, such as bleeding. Using a risk assessment model to predict the risk of blood clots could facilitate selection of patients for prophylaxis and optimise the balance of benefits, risks and costs. Objectives We aimed to identify validated risk assessment models and estimate their prognostic accuracy, evaluate the cost-effectiveness of different strategies for selecting hospitalised patients for prophylaxis, assess the feasibility of using efficient research methods and estimate key parameters for future research. Design We undertook a systematic review, decision-analytic modelling and observational cohort study conducted in accordance with Enhancing the QUAlity and Transparency Of health Research (EQUATOR) guidelines. Setting NHS hospitals, with primary data collection at four sites. Participants Medical and surgical hospital inpatients, excluding paediatric, critical care and pregnancy-related admissions. Interventions Prophylaxis for all patients, none and according to selected risk assessment models. Main outcome measures Model accuracy for predicting blood clots, lifetime costs and quality-adjusted life-years associated with alternative strategies, accuracy of efficient methods for identifying key outcomes and proportion of inpatients recommended prophylaxis using different models. Results We identified 24 validated risk assessment models, but low-quality heterogeneous data suggested weak accuracy for prediction of blood clots and generally high risk of bias in all studies. Decision-analytic modelling showed that pharmacological prophylaxis for all eligible is generally more cost-effective than model-based strategies for both medical and surgical inpatients, when valuing a quality-adjusted life-year at £20,000. The findings were more sensitive to uncertainties in the surgical population; strategies using risk assessment models were more cost-effective if the model was assumed to have a very high sensitivity, or the long-term risks of post-thrombotic complications were lower. Efficient methods using routine data did not accurately identify blood clots or bleeding events and several pre-specified feasibility criteria were not met. Theoretical prophylaxis rates across an inpatient cohort based on existing risk assessment models ranged from 13% to 91%. Limitations Existing studies may underestimate the accuracy of risk assessment models, leading to underestimation of their cost-effectiveness. The cost-effectiveness findings do not apply to patients with an increased risk of bleeding. Mechanical thromboprophylaxis options were excluded from the modelling. Primary data collection was predominately retrospective, risking case ascertainment bias. Conclusions Thromboprophylaxis for all patients appears to be generally more cost-effective than using a risk assessment model, in hospitalised patients at low risk of bleeding. To be cost-effective, any risk assessment model would need to be highly sensitive. Current evidence on risk assessment models is at high risk of bias and our findings should be interpreted in this context. We were unable to demonstrate the feasibility of using efficient methods to accurately detect relevant outcomes for future research. Future work Further research should evaluate routine prophylaxis strategies for all eligible hospitalised patients. Models that could accurately identify individuals at very low risk of blood clots (who could discontinue prophylaxis) warrant further evaluation. Study registration This study is registered as PROSPERO CRD42020165778 and Researchregistry5216. Funding This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: NIHR127454) and will be published in full in Health Technology Assessment; Vol. 28, No. 20. See the NIHR Funding and Awards website for further award information. Plain language summary People who are admitted to hospital are at risk of blood clots that can cause serious illness or death. Patients are often given low doses of blood-thinning drugs to reduce this risk. However, these drugs can cause side effects, such as bleeding. Hospitals currently use complex risk assessment models (risk scores, which usually include patient, disease, mobility and intervention factors) to determine the individual risk of blood clots and identify people most likely to benefit from blood-thinning drugs. There are a lot of different risk scores and we do not know which one is best. We also do not know how these scores compare to each other or whether using scores to decide who should get blood-thinning drugs provides good value for money to the NHS. We reviewed all previous studies of risk scores. We found that they did not predict blood clots very well and we could not recommend one score over another. We then created a mathematical model to simulate the use of blood-thinning drugs in people admitted to hospital. The model suggested that giving blood-thinning drugs to everyone who could have them would probably provide the best value for money, in medical patients. Our findings were the same, but less certain, for surgical patients. We also collected information from four NHS hospitals to explore possibilities for future research. Our work showed that routinely collected electronic data on blood clots and bleeding events is not very accurate and that using different scores could result in variable use of blood-thinning medications. Our findings suggest that it may be better value to the NHS and better for patients if we were to offer blood-thinning medications to everyone on admission to hospital, without using any risk score. However, this approach needs further research to ensure it is safe and effective. Such research would not be able to rely on routine electronic data to identify blood clots or bleeding events, in isolation. Scientific summary Background Venous thromboembolism (VTE) is a composite diagnosis including deep-vein thrombosis (DVT) and pulmonary embolism (PE). The condition is a global health burden, affecting over 1:1000 adults worldwide, every year. Previous epidemiological studies have shown more than half of all VTE events to be hospital-associated (or acquired) thrombosis (HAT), occurring during admission or within 90 days of discharge. Current evidence suggests that many HAT events are potentially preventable. Pharmacological prophylaxis in medical and surgical inpatients has been shown to be clinically effective in several large studies and meta-analyses. However, although prophylaxis reduces the risk of VTE for hospital inpatients, it also incurs costs and potentially increases the risk of bleeding. As such, international guidelines recommend a process of individualised VTE risk assessment at the point of hospital admission. Several risk assessment models (RAMs) have been developed to aid this process. The accuracy and cost-effectiveness of these models are uncertain. Further prospective studies in this area are challenging due to national guidance, contract standards and high rates of routine prescribing (> 70%). We sought to assess the clinical and cost-effectiveness of VTE RAMs through secondary research methods, by conducting a systematic review of RAM accuracy and subsequent decision-analytic modelling. In addition, we sought to evaluate the accuracy of routine data sources to identify core VTE and bleeding outcomes and estimate key clinical parameters for any future implementation study. Objectives The prespecified and combined project objectives were as follows: Update existing systematic reviews to identify VTE RAMs for hospital inpatients and determine their comparative accuracy for predicting the risk of VTE. Undertake decision-analytic modelling to determine the cost-effectiveness of pharmacological prophylaxis guided by a RAM, compared to prophylaxis for all and prophylaxis for none. Use the decision-analytic model to identify key areas of uncertainty and determine the value of gathering additional information to reduce uncertainty. Pilot the use and evaluate the accuracy of efficient methods to measure core clinical VTE and bleeding outcomes. Estimate key parameters for planning a future implementation study. Methods Workstream 1 used a systematic review and economic analysis to address objectives 1, 2 and 3. We extended and updated overlapping systematic reviews of available RAMs, performed quality assessment of relevant studies and synthesised performance measures (e.g. sensitivity, specificity and concordance (C) statistics) to evaluate the prognostic accuracy of individual RAMs. We subsequently used this information to inform decision-analytic modelling. The target population for the conceptual model was hospital inpatients, including medical, surgical and trauma patients but excluding critical care patients, children and women admitted to hospital for pregnancy-related reasons. Patients at increased risk of bleeding were assumed not to receive pharmacological prophylaxis under any strategy and were excluded from the model. We compared strategies of using no pharmacological prophylaxis, pharmacological prophylaxis for all and pharmacological prophylaxis given in accordance with validated RAMs. The use of mechanical prophylaxis was considered out of scope for the modelling. We used a lifetime horizon and reported primary outcomes of costs and quality-adjusted life-years (QALYs). Multiple scenario and sensitivity analyses were conducted by subgroup. Workstream two used primary research to address objectives four and five. We conducted a multicentre observational cohort study across four hospitals to compare prespecified methods of efficient data collection to formal case note review. We identified a target population hospitalised during 2019 and extracted data from prospectively completed VTE risk assessments, when available. Research assistants undertook further retrospective case note review for each patient episode. We used electronic health records to evaluate additional risk characteristics and record all relevant clinical outcomes, including the subsequent diagnosis of HAT, major bleeding and clinically relevant non-major bleeding events. We then collated multiple routine data sources for each patient episode to determine the accuracy of these methods in identifying relevant clinical outcomes, compared to a gold standard of case note review. The primary outcome measures for workstream two were contingency tables with sensitivity, specificity and predictive values. Feasibility criteria for any future study using efficient data methods were set a priori. We also estimated the potential variation in prescribing recommendations using multiple validated RAMs and estimated key parameters for any future implementation study. The Venous ThromboEmbolism Assessment Methods study received a favourable opinion from the Proportionate Review Sub-committee of the London – West London & Gene Therapy Advisory Committee (GTAC) Research Ethics Committee and approval from the Health Research Authority (HRA) and Care Research Wales (HCRW) on 18 September 2019 (reference 19/LO/1303, IRAS project ID 262220). Results Workstream 1 Our updated systematic review included 51 studies, comprising 24 unique validated RAMs. The vast majority of studies evaluated VTE RAMs in medical (n = 21), surgical (n = 15) or mixed (n = 4) cohorts of hospital inpatients. The most widely evaluated models were the Caprini RAM (22 studies), the Padua prediction score (16 studies), the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) models (8 studies), the Geneva Risk Score (4 studies) and the Kucher Score (4 studies). All studies had a high or unclear risk of bias, with the main issues related to patient selection, outcome and analysis factors. C-statistics varied markedly between these studies and between models, with no RAM performing obviously better than other models. Similarly, estimates for sensitivity and specificity were highly variable. In the decision analytic modelling, we estimate that in medical inpatients, prophylaxis reduces serious adverse outcomes [fatal PEs, fatal bleeding and non-fatal intracranial haemorrhage (ICH)] from 53 per 10,000 to 42 per 10,000, with a reduction in symptomatic DVTs and non-fatal PEs that is higher than the increase in other bleeds. In the long-term outcomes at 5 years, there is a large reduction in the number of patients experiencing post-thrombotic syndrome (PTS) following prophylaxis from 787 per 10,000 to 385 per 10,000, but minimal difference in overall survival. Modelling showed that thromboprophylaxis for all has a high probability (> 99%) of being the most cost-effective strategy (£20,000 per QALY threshold) based on the performance of existing RAMs in cohorts of medical inpatients. This finding was generally robust under the scenario and sensitivity analyses with one exception; targeting thromboprophylaxis using a Padua score ≥ 3 (84% of cohort) had a 76.6% probability of being the most cost-effective strategy when assuming higher RAM performance (sensitivity 99.9%; specificity 23.7%). The overall expected value of perfect information (EVPI) associated with all parameters included in the probabilistic sensitivity analysis (PSA) when valuing a QALY at £20,000 was £0.06 per patient per year due to the high probability that prophylaxis for all is the optimal strategy. However, the EVPI would be £2.42 per person per year if a more accurate RAM could be developed for medical inpatients. In surgical inpatients, the risk of a serious adverse outcome (as above) is low at 7 per 10,000 but it is increased slightly by prophylaxis to 11 per 10,000 due to the additional risk of fatal bleeding or non-fatal ICH. Although the risk of any symptomatic VTE is reduced form 140 per 10,000 to 41 per 10,000, the risk of any major bleeding is increased from 125 per 10,000 to 370 per 10,000. Prophylaxis still reduces the risk of PTS in surgical inpatients, from 367 per 10,000 to 107 per 10,000, but PTS is less common in the surgical cohort than in the medical cohort. Modelling showed that giving prophylaxis to all surgical inpatients has the highest probability of being cost-effective at £20,000 per QALY, with a 70% likelihood of being optimal, but there is also a 17% likelihood that using a Pannucci score of ≥ 3 would be optimal and a 9% likelihood that using a Pannucci score of ≥ 1 would be optimal. These findings were sensitive to some uncertainties explored in the scenario and sensitivity analyses. In particular, prophylaxis for all may no longer be the optimal strategy for surgical inpatients when the risk of PTS is lower, when using an extended duration of prophylaxis or where a RAM is assumed to have a very high sensitivity. The overall EVPI associated with all parameters included in the PSA when valuing a QALY at £20,000 was £16.35 per person. This does not include the uncertainty of considering which RAM to use as the EVPI and does not include any uncertainty not captured in the PSA, such as uncertainty around the model assumptions or the choice of data sources, or uncertainty around the estimates of sensitivity and specificity. In the analysis of parameter EVPI [expected value of partial perfect information (EVPPI)], the relative risk of VTE for patients having thromboprophylaxis compared to patients not having thromboprophylaxis was the parameter with the largest EVPPI. In specific surgical populations requiring longer durations of prophylaxis, we found that pharmacological prophylaxis for all eligible patients remained optimal in those having elective knee replacement. Offering prophylaxis at a Pannucci score of ≥ 3 (sensitivity of 84% and specificity of 49%) was optimal when prescribing 28 days of low-molecular-weight heparin (LMWH) to patients having elective hip replacement. These conclusions were sensitive to prophylaxis strategy; using direct oral anticoagulants (DOACs) rather than LMWH resulted in prophylaxis for all as the optimal strategy for both hip and knee patients. Workstream 2 The observational cohort study was conducted across all four sites, enrolling 2115 patient hospitalisation episodes, with 2008 eligible for analysis. Medical and surgical cases were evenly balanced, but with more emergency (73.7%) than elective (25.8%) admissions. The sensitivity of routine coding data for detection of HAT and major bleeding events was 62% [95% confidence interval (CI) 54 to 69] and 38% (95% CI 27 to 50), respectively. Local VTE data sets performed better, with sensitivity of 81% (95% CI 75 to 87). The specificity of routine coding data for VTE and bleeding was 98% (95% CI 97 to 99) and 95% (95% CI 94 to 96), respectively, and the specificity of local VTE data sets was 100% (95% CI 99 to 100). We were unable to demonstrate overall feasibility of using efficient outcome measures and did not meet several prespecified criteria. In a smaller subgroup of patients with prospectively collected data, we evaluated potential variation in pharmacological prophylaxis using different RAMs. We identified 543 hospital episodes with VTE risk assessment performed during 2019, with 254 episodes suitable for inclusion, data extraction and comparative analysis. Overall recommendations for pharmacological prophylaxis varied substantially between seven RAMs, ranging from 13% of admitted patients for the IMPROVE associative score (95% CI 9.4 to 17.7) to 91% (95% CI 86.3 to 93.6) for the Department of Health (now Department of Health and Social Care) VTE risk assessment tool. The latter tool resulted in an absolute increase of 28% for prophylaxis recommendation, compared to other RAMs. Within this multicentre cohort of hospitalised inpatients, the HAT event rate was 1.6% (95% CI 1.0 to 2.2) and the major bleeding event rate was 2.5% (95% CI 1.8 to 3.2). Conclusions We updated prior systematic reviews to conclude that that the available evidence has significant methodological limitations and demonstrates current RAMs to have weak predictive accuracy. Decision-analytic modelling showed that the balance of VTE or major bleeding risks, combined with the RAM performance in medical cohorts, means that thromboprophylaxis for all is the optimal strategy within our model for medical inpatients. These findings were robust to multiple scenarios and sensitivity analyses. The optimal strategy for surgical inpatients is to offer thromboprophylaxis to all. However, the scenario analyses for surgical inpatients found that the optimal strategy was sensitive to many of the individual model inputs and assumptions tested. In addition, the optimal strategy for surgical patients receiving extended duration thromboprophylaxis appeared dependent on duration of therapy and prophylaxis strategy. The findings from our cohort study suggest that efficient methods for identifying VTE or major bleeding events during hospital admission or within 90 days of discharge, are not sufficiently sensitive for use in a large data-enabled study. We did not reach several predefined feasibility metrics. We also found limitations in the ability of efficient methods to identify individual risk variables and facilitate RAM comparison in future work. The majority of our sites did not collect contemporaneous data on risk assessment in a digital, or easily accessible format. Implications for policy-makers If, despite drug costs and potential harms, pharmacological prophylaxis for all is the most cost-effective strategy, use of unvalidated RAMs may be suboptimal. Based on our findings, it may be preferable for policy-makers to consider evaluating a new paradigm of ‘opt-out’ VTE prevention, in which all eligible patients are routinely offered pharmacological prophylaxis on hospital admission without complex risk assessment. In this circumstance, patients who are ineligible for pharmacological prophylaxis due to high bleeding risk, specific contraindications or personal choice could opt out, but still utilise evidence-based mechanical thromboprophylaxis strategies. The findings from our cohort study have implications for funders looking to support further work in this area. These data suggest large studies entirely reliant on routine data collection methods for complex time-dependent outcome measures (such as HAT or major bleeding) are likely to be inaccurate. Recommendations for future research Further research should evaluate the following themes in this area of clinical care: Evaluation of routine pharmacological prophylaxis for all eligible medical and surgical patients on hospital admission, compared to current practice of risk assessment using a RAM. Such work would need to be conducted at scale and could not rely on efficient methods for outcome measurement in isolation. Development and validation of RAMs to identify individuals receiving prophylaxis at very low risk of blood clots, who could therefore potentially discontinue pharmacological prophylaxis early during hospital stay. Our analysis was inevitably limited in scope and did not examine the use of RAMs to identify accurate prediction of bleeding risk, the use of mechanical prophylaxis in medical patients with increased bleeding risk (or contraindications to pharmacological prophylaxis) or patients at risk of thromboprophylaxis failure. These issues could be priorities for future research. Study registration This study is registered as Research Registry 5216, PROSPERO CRD42020165778. Funding This award was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme (NIHR award ref: NIHR127454) and is published in full in Health Technology Assessment; Vol. 28, No. 20. See the NIHR Funding and Awards website for further award information.
Keywords
