BMJ Medicine (Aug 2025)
Reporting guideline for chatbot health advice studies: the Chatbot Assessment Reporting Tool (CHART) statement
- ,
- Elizabeth Loder,
- Xi Chen,
- Gordon Guyatt,
- Per Olav Vandvik,
- Thomas Agoritsas,
- Riaz Agha,
- Xiaoxuan Liu,
- Yung Lee,
- Alfonso Iorio,
- Monica Ortenzi,
- Michael Mittelman,
- Joerg Meerpohl,
- Gary Collins,
- Michael Berkwits,
- Annette Flanagin,
- Nan Liu,
- Cynthia Lokker,
- Nipun Verma,
- Ashirbani Saha,
- Julio Mayol,
- Michael Anderson,
- Hugh Harvey,
- Eliseo Guallar,
- Stavros A Antoniou,
- Melissa McCradden,
- Piyush Mathur,
- Xiaomei Yao,
- Carolyn Canfield,
- Christine Laine,
- Stacy Loeb,
- Timothy Feeney,
- Gregor Štiglic,
- Tyler McKechnie,
- Bright Huo,
- David Chartash,
- Jeremy Y Ng,
- Diana Samuel,
- Helen Frankish,
- Karim Ramji,
- Vanessa Boudreau,
- Giovanni Cacciamani,
- Daniela Pacella,
- Jennifer C Camaradou,
- Anthony Sunjaya,
- Arun Thirunavukarasu,
- An Wen Chan
Affiliations
- Elizabeth Loder
- professor
- Xi Chen
- 14 Department of Bioengineering, Imperial College London, London, UK
- Gordon Guyatt
- distinguished professor
- Per Olav Vandvik
- professor
- Thomas Agoritsas
- associate professor
- Riaz Agha
- Xiaoxuan Liu
- associate professor
- Yung Lee
- 2 Michael G. DeGroote School of Medicine, McMaster University, Hamilton, Ontario, Canada
- Alfonso Iorio
- professor
- Monica Ortenzi
- King’s College Hospital, London, UK
- Michael Mittelman
- Philadelphia, Pennsylvania, USA
- Joerg Meerpohl
- 30 Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany
- Gary Collins
- professor of medical statistics
- Michael Berkwits
- director
- Annette Flanagin
- executive managing editor
- Nan Liu
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore
- Cynthia Lokker
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
- Nipun Verma
- Hepatology, Post Graduate Institute of Medical Education and Research (PIGMER), Chandigarh, India
- Ashirbani Saha
- 2 Injury Prevention Research Office, St. Michael’s Hospital, Toronto, Ontario, Canada
- Julio Mayol
- 13 Surgery, Hospital Clinico Universitario San Carlos, Madrid, Madrid, Spain
- Michael Anderson
- Department of Health Policy, The London School of Economics and Political Science, London, UK
- Hugh Harvey
- Eliseo Guallar
- Departments of Epidemiology and Medicine, and Welch Center for Prevention, Epidemiology, and Clinical Research, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA
- Stavros A Antoniou
- Mediterranean Hospital of Cyprus, Limassol, Cyprus
- Melissa McCradden
- Piyush Mathur
- staff anaesthesiologist
- Xiaomei Yao
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
- Carolyn Canfield
- 3 Department of Family Practice, The University of British Columbia Faculty of Medicine, Vancouver, British Columbia, Canada
- Christine Laine
- senior deputy editor, Annals of Internal Medicine
- Stacy Loeb
- Urology, Population Health, New York University, New York, New York, USA
- Timothy Feeney
- researcher
- Gregor Štiglic
- Faculty of Health Sciences, University of Maribor, Maribor, Slovenia
- Tyler McKechnie
- Department of Health Research Methodology, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
- Bright Huo
- David Chartash
- Jeremy Y Ng
- Diana Samuel
- Helen Frankish
- Karim Ramji
- Vanessa Boudreau
- Giovanni Cacciamani
- Daniela Pacella
- Jennifer C Camaradou
- Anthony Sunjaya
- Arun Thirunavukarasu
- An Wen Chan
- DOI
- https://doi.org/10.1136/bmjmed-2025-001632
- Journal volume & issue
-
Vol. 4,
no. 1
Abstract
The Chatbot Assessment Reporting Tool (CHART) is a reporting guideline developed to provide reporting recommendations for studies evaluating the performance of generative artificial intelligence (AI)-driven chatbots when summarising clinical evidence and providing health advice, referred to as chatbot health advice studies. CHART was developed in several phases after performing a comprehensive systematic review to identify variation in the conduct, reporting, and method in chatbot health advice studies. Findings from the review were used to develop a draft checklist that was revised through an international, multidisciplinary, modified, asynchronous Delphi consensus process of 531 stakeholders, three synchronous panel consensus meetings of 48 stakeholders, and subsequent pilot testing of the checklist. CHART includes 12 items and 39 subitems to promote transparent and comprehensive reporting of chatbot health advice studies. These include title (subitem 1a), abstract/summary (subitem 1b), background (subitems 2a,b), model identifiers (subitems 3a,b), model details (subitems 4a-c), prompt engineering (subitems 5a,b), query strategy (subitems 6a-d), performance evaluation (subitems 7a,b), sample size (subitem 8), data analysis (subitem 9a), results (subitems 10a-c), discussion (subitems 11a-c), disclosures (subitem 12a), funding (subitem 12b), ethics (subitem 12c), protocol (subitem 12d), and data availability (subitem 12e). The CHART checklist and corresponding diagram of the method were designed to support key stakeholders including clinicians, researchers, editors, peer reviewers, and readers in reporting, understanding, and interpreting the findings of chatbot health advice studies.
