Application of an interactive diagnosis ranking algorithm in a simulated vignette-based environment for general dermatology

Abstract

Background: Diagnostic algorithms may reduce noise and bias and improve interrater agreement of clinical decisions. In a practical sense, algorithms may serve as alternatives to specialist consultations or decision support in store-and-forward teledermatology. It is, however, unknown how dermatologists interact with algorithms based on questionnaires. Objective: To evaluate the performance of a questionnaire-based diagnostic algorithm when applied by users with different expertise. Methods: We created 58 virtual test cases covering common dermatologic diseases and asked five raters with different expertise to complete a predefined clinical questionnaire, which served as input for a disease ranking algorithm. We compared the ranks of the correct diagnosis between users, analysed the similarity between inputs of different users, and explored the impact of different parts of the questionnaire on the final ranking. Results: When applied by a board-certified dermatologist, the algorithm top-ranked the correct diagnosis in the majority of cases (median rank 1; IQR: 1.0; mean reciprocal rank 0.757). The median rank of the correct diagnosis was significantly lower when the algorithm was applied by four dermatology residents (median rank 2-5, p<.01 for all). The lowest similarity between inputs of the residents and the board-certified dermatologist was found for questions regarding morphology. Sensitivity analysis showed the highest deterioration in performance after omission of information on morphology and anatomic site. Conclusions: A simple questionnaire-based disease ranking algorithm provides accurate ranking for a wide variety of dermatologic conditions. When applied in clinical practice, additional measures may be needed to ensure robustness of data entry for inexperienced users.

Keywords

• logic
• diagnosis
• algorithm
• ranking
• artificial intelligence
• dermtrainer