Language models in digital psychiatry: challenges with simplification of healthcare materials

Abstract

Abstract Linguistic hurdles in healthcare, such as complex language, significantly affect patient outcomes, including satisfaction with interaction, comprehension of healthcare materials, and engagement with the healthcare system. Reducing these hurdles has been a focus in healthcare delivery, as they significantly hinder patient engagement and adherence to treatments. The growing use of large language models (LLMs) in healthcare opens the possibility to reduce these linguistic hurdles. This study evaluates the ability of five prominent LLMs—GPT-3.5, GPT-4, GPT-4o, LLaMA-3, and Mistral—to simplify healthcare information to the standard recommended by the American Journal of Medicine. Our results indicate that while LLMs can approximate targeted reading levels, their outputs are inconsistent, with significant variability in reading level and deviation from the topic, making them unsuitable for deployment in healthcare settings.