Health Technology Assessment (Jan 2024)
Undertaking Studies Within A Trial to evaluate recruitment and retention strategies for randomised controlled trials: lessons learnt from the PROMETHEUS research programme
Abstract
Background Randomised controlled trials (‘trials’) are susceptible to poor participant recruitment and retention. Studies Within A Trial are the strongest methods for testing the effectiveness of strategies to improve recruitment and retention. However, relatively few of these have been conducted. Objectives PROMoting THE Use of Studies Within A Trial aimed to facilitate at least 25 Studies Within A Trial evaluating recruitment or retention strategies. We share our experience of delivering the PROMoting THE Use of Studies Within A Trial programme, and the lessons learnt for undertaking randomised Studies Within A Trial. Design A network of 10 Clinical Trials Units and 1 primary care research centre committed to conducting randomised controlled Studies Within A Trial of recruitment and/or retention strategies was established. Promising recruitment and retention strategies were identified from various sources including Cochrane systematic reviews, the Study Within A Trial Repository, and existing prioritisation exercises, which were reviewed by patient and public members to create an initial priority list of seven recruitment and eight retention interventions. Host trial teams could apply for funding and receive support from the PROMoting THE Use of Studies Within A Trial team to undertake Studies Within A Trial. We also tested the feasibility of undertaking co-ordinated Studies Within A Trial, across multiple host trials simultaneously. Setting Clinical trials unit-based trials recruiting or following up participants in any setting in the United Kingdom were eligible. Participants Clinical trials unit-based teams undertaking trials in any clinical context in the United Kingdom. Interventions Funding of up to £5000 and support from the PROMoting THE Use of Studies Within A Trial team to design, implement and report Studies Within A Trial. Main outcome measures Number of host trials funded. Results Forty-two Studies Within A Trial were funded (31 host trials), across 12 Clinical Trials Units. The mean cost of a Study Within A Trial was £3535. Twelve Studies Within A Trial tested the same strategy across multiple host trials using a co-ordinated Study Within A Trial design, and four used a factorial design. Two recruitment and five retention strategies were evaluated in more than one host trial. PROMoting THE Use of Studies Within A Trial will add 18% more Studies Within A Trial to the Cochrane systematic review of recruitment strategies, and 79% more Studies Within A Trial to the Cochrane review of retention strategies. For retention, we found that pre-notifying participants by card, letter or e-mail before sending questionnaires was effective, as was the use of pens, and sending personalised text messages to improve questionnaire response. We highlight key lessons learnt to guide others planning Studies Within A Trial, including involving patient and public involvement partners; prioritising and selecting strategies to evaluate and elements to consider when designing a Study Within A Trial; obtaining governance approvals; implementing Studies Within A Trial, including individual and co-ordinated Studies Within A Trials; and reporting Study Within A Trials. Limitations The COVID-19 pandemic negatively impacted five Studies Within A Trial, being either delayed (n = 2) or prematurely terminated (n = 3). Conclusions PROMoting THE Use of Studies Within A Trial significantly increased the evidence base for recruitment and retention strategies. When provided with both funding and practical support, host trial teams successfully implemented Studies Within A Trial. Future work Future research should identify and target gaps in the evidence base, including widening Study Within A Trial uptake, undertaking more complex Studies Within A Trial and translating Study Within A Trial evidence into practice. Study registration All Studies Within A Trial in the PROMoting THE Use of Studies Within A Trial programme had to be registered with the Northern Ireland Network for Trials Methodology Research Study Within A Trial Repository. Funding This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: 13/55/80) and is published in full in Health Technology Assessment; Vol. 28, No. 2. See the NIHR Funding and Awards website for further award information. Plain language summary Aims of the research A Study Within A Trial is a research study nested inside a larger ‘host trial’, promoting the use of Studies Within A Trial aimed to do Study Within A Trial routine practice in clinical trial units by funding and supporting at least 25 Studies Within A Trial. Background The best way to test health and social care treatments is to do a randomised controlled trial (‘trial’), where some patients get the treatment being tested and some do not. The results of different groups are compared to see if the treatment improves care. Recruiting patients and keeping them involved in trials is often very difficult. Research teams often do not know how best to recruit and keep patients engaged as the methods have not been tested to see if they work. The best way to test these methods is by doing a Study Within A Trial. We test a programme of Studies Within A Trial for recruiting and keeping patients engaged in trials. Methods Trial teams were able to apply for funding of up to £5000 and receive support from Promoting the use of Study Within A Trial team to do Studies Within A Trial. We used our experience of doing Studies Within A Trial to outline lessons learnt for doing Studies Within A Trial. Results We funded 42 Studies Within A Trial and gave teams necessary advice to do them. We significantly increased the knowledge for both recruitment and retention strategies, and found ‘pre-notifying’ before sending questionnaires, sending pens and personalised text messages were all effective for increasing responses by participants. We tested Studies Within A Trial across several different trials at the same time to find out more quickly whether their methods worked. We highlight key lessons learnt to guide others doing Studies Within A Trial, including involving patient partners; picking the right strategy to test; getting ethical approvals; how to do and report Studies Within A Trial. Conclusions Promoting the use of studies within a trial was successful and supported more Studies Within A Trial than planned. We hope our experience will support those doing Studies Within A Trial in the future. Scientific summary Background Many delivery aspects of randomised controlled trials (RCTs) have not been subjected to rigorous evaluation. Strategies to enhance recruitment and retention often tend to not be based on evidence. However, there is an increasing interest in embedding RCTs of such strategies. The Medical Research Council (MRC) funded systematic techniques for assisting recruitment to trials (START), which found it feasible to test two strategies for recruitment and retention across a number of trials by performing randomised ‘Studies Within A Trial (SWATs)’. Aims The aim of the PROMoting THE USE of Studies Within A Trial (PROMETHEUS) programme was to build on the START initiative and make embedding SWATs standard practice across Clinical Trials Units (CTUs). We intended to pump prime and facilitate the start of at least 25 SWATs across multiple CTUs within 30 months. The ultimate aim was to make the inclusion of SWATs routine when conducting a trial in a CTU. We share our experience of delivering the PROMETHEUS programme, along with the lessons learnt for undertaking randomised SWATs of recruitment and retention methods. Methods A network of 10 CTUs and one primary care research centre committed to conducting randomised controlled SWATs of recruitment and/or retention interventions was established by the PROMETHEUS team. We identified promising recruitment and retention interventions from a variety of sources including Cochrane systematic reviews, the Northern Ireland Network for Trial Methodology Research SWAT Repository Store, and existing prioritisation exercises. Promising strategies were reviewed by our patient and public (PPI) members to create an initial priority list of seven recruitment and eight retention interventions. The programme allowed host trial teams to apply for funding of up to £5000 and receive support from the PROMETHEUS team to design, implement and report SWATs. We also tested the feasibility of undertaking co-ordinated SWATs, across multiple host trials simultaneously. Results The PROMETHEUS programme funded 42 SWATs, embedded within 31 host trials, across 12 CTUs. The mean cost of a funded SWAT was £3535. Of the 42 SWATs, 12 tested the same SWAT across multiple host trials using a co-ordinated SWAT design and four tested more than one strategy in a factorial design. Two recruitment and five retention strategies were evaluated in more than one host trial. In the space of just 4 years, PROMETHEUS will add 18% more SWATs to the Cochrane review of global recruitment strategies, and 79% more SWATs to the Cochrane review of global retention strategies. The results from the SWATs reported to date found there was no evidence of a significant difference in recruitment for any of the strategies tested. For retention, pre-notifying participants by card prior to sending questionnaires was effective [risk difference 3.3%, 95% confidence interval (CI) −3.0% to 9.6%]; as was pre-notifying participants by letter or e-mail (risk difference 3.8%, 95% CI −6.1% to 13.6%). Sending personalised text messages was more effective for improving the return of postal questionnaires compared to non-personalised text messages (risk ratio 1.16, 95% CI 1.00 to 1.33); and resulted in fewer completions via telephone compared with a non-personalised text [adjusted odds ratio (OR) 0.44, 95% CI 0.22 to 0.87]. Including a pen with a questionnaire probably increases retention and response rate (pooled OR 1.21, 95% CI 1.09 to 1.35). We highlight key lessons learnt below (see Recommendations section). Conclusions The PROMETHEUS programme significantly increased the international evidence base for both recruitment and retention strategies within RCTs. The funded SWATs evaluated a wide range of recruitment and retention strategies; however, the COVID-19 pandemic negatively impacted five funded SWATs, with two being delayed and three prematurely terminated. Through this project, we identified that when provided with both funding and practical support, host trial teams successfully implemented SWATs. PROMETHEUS led to an overall increase in the evidence base; however, ongoing ‘routine’ application of SWATs across RCTs employing the lessons learnt is required to ensure that efficient trial conduct strategies are identified. Recommendations Recommendations for funders All trial funders should contribute to the effort to improve the efficiency of trials. Funders should encourage the teams that they fund to undertake SWATs. Funding streams specifically designed to support SWATs must be made available to trial teams to continue building the trial process evidence base, for recruitment and retention as well as for other stages of the trial design and delivery process. This includes funding streams for undertaking specific SWATs, as well as infrastructure funding to support CTUs and other centres to undertake co-ordination activities that will support the design, conduct, reporting, and implementation of SWATs and their findings to inform the work of the National Institute for Health and Care Research (NIHR), the MRC, and other funders. PROMETHEUS has demonstrated that co-ordination of activity remains crucial to the delivery of SWATs. A central, national co-ordination point that provides hands-on support needs to continue and funding should be allocated for this. Additionally, CTUs should identify a lead for SWATs, to support SWAT activities and evidence-based trial conduct within the CTU, as well as links with others undertaking SWATs elsewhere to share best practice. The funding for both central and CTU-based support should be ongoing. SWAT priorities need to be identified and communicated clearly to funders, and funders should use these priorities to inform their funding decisions. Funders should develop a mechanism to promote SWAT questions that have been identified as a priority during the funding application process. The mean cost of funding requested for a standalone SWAT within PROMETHEUS was £3535 (range £500–5000). The co-ordinated SWATs cost was £10,668 (training SWAT) and £1306.40 (Christmas cards); however, these did not include costs for central co-ordination, data preparation and sharing by the host trial teams, data cleaning, analysis and write-up. These costs suggest that the £10,000 being offered by the NIHR for trial teams to include a SWAT should be sufficient for most planned SWATs. However, there may be occasions where trial teams may wish to test strategies that may be more expensive. When applying for SWAT funding, trial teams should be asked to indicate whether the question they are addressing is a priority SWAT question, and to provide a clear rationale for selecting that particular question. If teams are unable to undertake a SWAT, funders should ask that recruitment and retention methods are clearly reported to support the evidence base. Recommendations for Sponsors Our experience suggests that there is a need for clear, easily accessible information about the nature of SWATs, as well as the role of the funder in supporting SWATs. Any future changes proposed by the Health Research Authority (HRA) to the approvals process need to be communicated clearly and applied consistently to each SWAT. Recommendations for involving patient and public involvement partners in Studies Within A Trial research PPI should be considered when undertaking a SWAT, in the same way PPI is expected to be undertaken in the main trial. PPI partners should be involved to develop novel and untested recruitment and retention strategies, as well as to adapt existing strategies to the context of their specific host trial and the population being enrolled. Recommendations for oversight committees Our experience suggests that Trial Management Groups (TMGs) play a key role in decisions about whether a SWAT is undertaken and continued in the host trial or not. TMG members should encourage the uptake of SWATs in their trials. While the findings of SWATs may not always directly inform their host trial, the findings of SWATs undertaken during the early, or internal pilot, phase of the trial may inform the decisions about which strategies should be used at a later stage, such as in the main trial. Trial Steering Committees should review the SWAT activity and progress, in the same way that they review substudies in a trial. Data Monitoring Committee review is dependent on the specific host trial and SWAT strategy being evaluated. Recommendations for journals and reviewers Journal peer reviewer profiles should be updated to include methodological interests and expertise, to support evidence of suitability to undertake a peer review for a SWAT. When selecting peer reviewers, the SWAT and methodology interests as registered by reviewers should be used where possible. SWATs are a niche area and so to increase the pool of reviewers, journals should consider being more flexible when assessing reviewer credentials to review a SWAT, such as allowing relevant experience in place of a PhD. Reviewers should be advised that in many instances informed consent from participants need not be obtained when undertaking a SWAT, due to the low risk associated with the intervention and in the case of retention SWATs due to existing consent for further research being in place. This may depend on the jurisdiction. In the UK, this approach is supported by HRA guidance in relation to grading for SWAT interventions and approvals. Robust and transparent reporting is necessary, that is compliant with CONsolidated Standards Of Reporting Trials (CONSORT). Recommendations for trial teams and methodologists undertaking SWAT research There remains a need for continually updated research priorities to allow researchers to address the questions relevant at that time. When SWAT priorities are set, methodologists need to provide enough information to enable teams to make informed decisions about evaluating the priorities set. Further work is needed to help teams identify suitable SWAT strategies for their host trials. SWAT priorities need to be communicated clearly and consistently to trial teams. Recommendation for trial conduct and using SWAT evidence As the evidence base develops for effective and cost-effective recruitment and retention strategies, it will become increasingly important for trial teams to use this evidence base to inform their recruitment and retention activities. Trial teams need to actively engage with the evidence base to inform their practice. Funders will need to actively support the trials they fund to use evidence-informed recruitment and retention strategies. Recommendations for future research There remains a substantial need for more high-quality SWAT evidence and so Chief Investigators should be encouraged to consider the embedding of a SWAT at the funding stage. Further work is therefore needed to increase the awareness of the methodological importance of SWAT research with research teams, and to develop engagement strategies to increase SWAT activity. Future research needs to focus on identifying whether further replications are needed for existing evidence. If so, the gaps in the evidence base should be targeted. More co-ordination and replication of SWAT evaluations are encouraged. A ‘real-time’ and dynamic communication strategy including a clear cost and resource breakdown for each suggested SWAT should be developed. This will alleviate the burden on trials teams to begin costing exercises and enable them to make an informed decision more quickly as to whether they can embed a given SWAT. There is a need to aid teams to identify and select a suitable SWAT for their host trial populations. Pragmatic decisions on which SWAT may be appropriate and feasible to include should be taken as required. A mechanism to communicate SWAT research priorities is needed, and this information needs to be readily accessible for all trialists to refer to. Our findings demonstrate that within an individual host trial, there is often a capacity to address more than one SWAT question, either separately, or simultaneously using a factorial design. This suggests that there is a capacity to significantly speed up and strengthen the evidence base through teams undertaking more than one SWAT in their trials where relevant. For certain strategies, co-ordinated SWATs should be encouraged. This method could be used to rapidly replicate SWAT evaluations to plug the evidence gap, as well as to evaluate more complex recruitment and retention strategies that may be more challenging to undertake using individual SWATs. Materials should be developed to advise teams on how to undertake co-ordinated SWATs, as well as a method of networking to enable teams to promote their co-ordinated SWAT and collaborate. As the evidence base develops, it will become increasingly important for trialists to utilise the evidence base in a systematic way to identify both effective and ineffective strategies to inform their practice. Future work should therefore consider issues around the dissemination and implementation of SWATs and develop guidance to enable the wider trials community to undertake, report and adopt the findings of SWATs. Implementation science, the study of methods to promote the uptake of evidence-based practice, could be used to inform any such future work. Funders can also help by questioning strategies proposed by trial teams that are known to be either ineffective or not cost-effective. Improving the knowledge of the potential ‘harms’ from implementing interventions that have no evidence of benefit is an important next step to help improve uptake. While establishing the effectiveness of recruitment or retention strategies is important, the high costs of research waste and limited public finance mean that cost considerations around SWATs are just as important. With only one retention strategy having high Grading of Recommendations, Assessment, Development and Evaluations (GRADE) certainty of cost effectiveness, we encourage trial teams to undertake streamlined economic evaluations alongside all their SWATs in the future, for strategies shown to be effective, as well as those that are ineffective. For cost effectiveness, trial teams should look to report the cost per additional participant recruited or retained (i.e. the incremental cost-effectiveness ratio). Value of information analyses can help determine the need for further SWAT evidence where several SWATs already exist. Many trial teams wish to contribute to developing the evidence base by undertaking non-randomised SWATs. Future work that informs the development of guidance for undertaking non-randomised SWATs would be helpful. Working with trial teams to develop engagement strategies and training to undertake SWATs would be beneficial. Audience specific guidance should be developed to support SWAT research. Trial teams have expressed they want to undertake SWATs that are important and necessary to increase the evidence base. Collaboration with funders, working groups involved in priority setting, and trial teams is needed to develop a mechanism to communicate this dynamic and evolving information once priority SWATs have been identified. Work is needed to identify the barriers that teams have when undertaking a SWAT, and strategies and solutions for addressing these barriers should be identified and implemented. Continued and proactive collaboration is needed with working groups to enable networking, and collaboration with teams undertaking SWAT research. Reporting guidance is needed to support teams when writing publications to ensure sufficient information is included to GRADE evaluation can be undertaken. Study registration All SWATs in the PROMETHEUS programme had to be registered with the Northern Ireland Network for Trials Methodology Research SWAT Repository. Funding details This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: 13/55/80) and is published in full in Health Technology Assessment; Vol. 28, No. 2. See the NIHR Funding and Awards website for further award information.
Keywords
