Exploring the feasibility and validity of a pragmatic approach to estimating the impact of long-term care: The 'expected' ASCOT method

Abstract

Context: Measuring the impact of long-term care (LTC) is essential if we are to allocate limited resources effectively.Objectives: We explored the feasibility and validity of a pragmatic approach to evaluation, known as the counterfactual self-estimation of programme participants (CSEPP). CSEPP forms part of the Adult Social Care Outcomes Toolkit (ASCOT), and is referred to as the ‘expected’ method since participants estimate their expected quality of life (QoL) in the absence of services.Methods: We used survey data from interviews with 748 LTC users in 22 English local authorities, which included questions on self- and interviewer-assessed understanding of the ‘expected’ questions. We used these data to assess feasibility. Construct validity was assessed by examining hypothesised associations between the expected score and individual characteristics. Bias was assessed by comparing the CSEPP impact estimate to one produced using the instrumental variables approach used by Forder et al. (2016, 2018) on the same dataset.Findings: We found evidence that the CSEPP/‘expected’ method was feasible and the self-estimated counterfactual outcome scores valid. There were indications that the method is less appropriate for some groups and that it may slightly overestimate the impact of LTC.Limitations: The main limitation is that the between methods comparison assumes that the instrumental variables approach provides a largely unbiased estimate of the effect of LTC, which is unlikely to be the case.Implications: The CSEPP/‘expected’ method is a useful tool in the LTC context, but more research is needed to understand potential sources of bias and its feasibility with certain groups.

Keywords

• long-term care
• ascot
• counterfactual
• self-estimation bias
• impact evaluation
• treatment effect