BMJ Open (May 2024)

Protocol for the development of the Chatbot Assessment Reporting Tool (CHART) for clinical advice

  • ,
  • Elizabeth Loder,
  • Gary S Collins,
  • Iain J Marshall,
  • Gordon Guyatt,
  • Per Olav Vandvik,
  • Joerg J Meerpohl,
  • Yaolong Chen,
  • David Moher,
  • An-Wen Chan,
  • Thomas Agoritsas,
  • Riaz Agha,
  • Yung Lee,
  • Alfonso Iorio,
  • Monica Ortenzi,
  • Farid Foroutan,
  • Michael Berkwits,
  • Annette Flanagin,
  • Cynthia Lokker,
  • Ashirbani Saha,
  • Julio Mayol,
  • Hugh Harvey,
  • Stavros A Antoniou,
  • Piyush Mathur,
  • Carolyn Canfield,
  • Christine Laine,
  • Stacy Loeb,
  • Timothy Feeney,
  • Hugo Campos,
  • Tyler McKechnie,
  • Bright Huo,
  • David Chartash,
  • Jeremy Y Ng,
  • Diana Samuel,
  • Helen Frankish,
  • Xufei Lao,
  • Karim Ramji,
  • Hassaan Ahmed,
  • Vanessa Boudreau

DOI
https://doi.org/10.1136/bmjopen-2023-081155
Journal volume & issue
Vol. 14, no. 5

Abstract

Read online

Introduction Large language model (LLM)-linked chatbots are being increasingly applied in healthcare due to their impressive functionality and public availability. Studies have assessed the ability of LLM-linked chatbots to provide accurate clinical advice. However, the methods applied in these Chatbot Assessment Studies are inconsistent due to the lack of reporting standards available, which obscures the interpretation of their study findings. This protocol outlines the development of the Chatbot Assessment Reporting Tool (CHART) reporting guideline.Methods and analysis The development of the CHART reporting guideline will consist of three phases, led by the Steering Committee. During phase one, the team will identify relevant reporting guidelines with artificial intelligence extensions that are published or in development by searching preprint servers, protocol databases, and the Enhancing the Quality and Transparency of health research Network. During phase two, we will conduct a scoping review to identify studies that have addressed the performance of LLM-linked chatbots in summarising evidence and providing clinical advice. The Steering Committee will identify methodology used in previous Chatbot Assessment Studies. Finally, the study team will use checklist items from prior reporting guidelines and findings from the scoping review to develop a draft reporting checklist. We will then perform a Delphi consensus and host two synchronous consensus meetings with an international, multidisciplinary group of stakeholders to refine reporting checklist items and develop a flow diagram.Ethics and dissemination We will publish the final CHART reporting guideline in peer-reviewed journals and will present findings at peer-reviewed meetings. Ethical approval was submitted to the Hamilton Integrated Research Ethics Board and deemed “not required” in accordance with the Tri-Council Policy Statement (TCPS2) for the development of the CHART reporting guideline (#17025).Registration This study protocol is preregistered with Open Science Framework: https://doi.org/10.17605/OSF.IO/59E2Q.