Assessing Artificial Intelligence Solution Effectiveness: The Role of Pragmatic Trials

Abstract

The emergence of artificial intelligence (AI) and other digital solutions in health care has considerably altered the landscape of medical research and patient care. Rigorous evaluation in routine practice settings is fundamental to the ethical use of AI and consists of 3 stages of evaluations: technical performance, usability and acceptability, and health impact evaluation. Pragmatic trials often play a key role in the health impact evaluation. The current review introduces the concept of pragmatic trials, their role in AI evaluation, the challenges of conducting pragmatic trials, and strategies to mitigate the challenges. We also examined common designs used in pragmatic trials and highlighted examples of published or ongoing AI trials. As more health systems advance into learning health systems, where outcomes are continuously evaluated to refine processes and tools, pragmatic trials embedded into everyday practice, leveraging data and infrastructure from delivering health care, will be a critical part of the feedback cycle for learning and improvement.