314 Large Language Model Approaches to Understand Differences Between Guidelines and Clinician Perception of Best Practices

Abstract

OBJECTIVES/GOALS: The Clinical Implementation stage in the translational pipeline is hampered by the tension between formal evidence and clinician perceptions. For instance, when guidelines are translated into electronic clinical decision support alerts, they are often ignored. Using advances in LLMs we present a framework to quantify these discrepancies. METHODS/STUDY POPULATION: We hypothesize that ignoring guideline-based alerts may be driven by discordances between clinical guidelines’ deterministic realities and clinician’ perception of clinical reality. Until now this has been very difficult to measure using quantitative methods. We argue that advances in Large Language Models (LLM) provide an avenue for exploring this quantitatively. Here we present the method and preliminary results comparing the responses of BioBERTT from a carefully designed set of questions when the LLM is fine-tuned using either formal guidelines or transcripts of clinicians discussing guidelines and clinical care in the parallel domain. The formal “distance” between the LLM responses is evaluated using quantitative metrics like the Hamming Distance. RESULTS/ANTICIPATED RESULTS: We present a description of the architecture used to prove or disprove our hypothesis. We will present results obtained when training the architecture with data that could be used to test the limits of our hypothesis, by fine-tuning BioBERT with diverse synthetic clinical views, either in agreement or disagreement with the formal guidelines. Results comparing sepsis guideline text with transcripts of interviews with Emergency Department clinicians discussing care practices for sepsis in the ED transcripts will also be considered. Our current emphasis is on securing a wider range of transcripts of clinicians interviewed from different clinical specialties and different clinical settings. While here we focus on clinical guidelines, the framework supports any intervention in the Clinical Implementation stage. DISCUSSION/SIGNIFICANCE: Leveraging recent advances in LLMs, we develop a framework that can quantitatively measure the differences between guidelines and clinician perception of best practices. We demonstrated the functionality of this approach using synthetic data and initiated the collection of clinician transcripts to test the framework in real clinical situations.