Use of the Behavior Assessment Tool in 18 Pilot Residency Programs

Abstract

Update. This article was updated on March 17, 2022, because of previous errors on pages 1, 4, 5, 6, and 7. In the Abstract, Results, and Discussion sections, the specificity of the ABOS Behavior Assessment Tool that had read “57%” and “57% (95% CI 52% to 62%)” now reads “51%” and “51% (95% CI 45% to 56%).” In the Abstract, “1,012 evaluators” now reads “1,016 evaluators” and “431 residents” now reads “428 residents.” In the Results section entitled “Evaluation Results per Resident,” the sentence that had read “The domain with the greatest number of residents exhibiting low scores was ethical behavior.” now reads “The domain with the greatest number of residents exhibiting low scores was interaction.” In the Discussion section on pages 6 and 7, the number of residents with low scores in at least one domain that had read “176” now reads “196.” In Table III, the title that had read “Behavior Evaluations Completed by Resident Year in Training Source” now reads “Behavior Evaluations Completed by Resident Year in Training.” In Table VII, the column head that had read “No. of Residents with >2 Low Scores within the Same Domain” now reads “No. of Residents with ≥2 Low Scores within the Same Domain.” Also in Table VII, in columns 2 and 3, the values that had read “26 (6%), 23 (5%), 19 (4%), 18 (4%), and 20 (5%)” now read “32 (8%), 61 (14%), 63 (15%), 57 (13%), and 59 (14%).” In Table VIII, the column head that had read “No. of Baseline Professionalism PD Assessment Low Score Residents (N = 32) Also with Low Scores on the Behavior Tool by at least 2 Evaluators” now reads “No. of Baseline Professionalism PD Assessment Low Score Residents (N = 32) Also with Low Scores on the Behavior Tool.” In Table IX, the title that had read “Number of Low Domain Scores by at least 2 Evaluators for Low Baseline Professionalism PD Assessment Score Residents (n = 32)*” now reads “Number of Low Domain Scores for Low Baseline Professionalism PD Assessment Score Residents (n = 32)*”. Also, the values in the table that had read “6, 3, 2, 2, 6, 13” now read “7, 2, 1, 3, 5, 14.” In Table X, the title that had read “Specificity and Sensitivity of the ABOS Behavior Tool Compared with PD Baseline Assessment for All Participating Residents (n = 440)*” now reads “Specificity and Sensitivity of the ABOS Behavior Tool Compared with PD Baseline Assessment for All Participating Residents (n = 428)*.” Also in Table X, in the right column entitled “PD Baseline Assessment High Score (3 or 4),” the values that had read “176” and “232” now read “196” and “200,” respectively. Finally, a supplementary data file has now been included with the article that contains results that portray resident performance with at least two low scores in one domain by at least two different evaluators. An erratum has been published: JBJS Open Access. 2022 Mar 31;7(1):e20.00103ER. Background:. The purpose of this study was to determine the feasibility and evaluate the effectiveness of the American Board of Orthopaedic Surgery Behavior Tool (ABOSBT) for measuring professionalism. Methods:. Through collaboration between the American Board of Orthopaedic Surgery and American Orthopaedic Association's Council of Residency Directors, 18 residency programs piloted the use of the ABOSBT. Residents requested assessments from faculty at the end of their clinical rotations, and a 360° request was performed near the end of the academic year. Program Directors (PDs) rated individual resident professionalism (based on historical observation) at the outset of the study, for comparison to the ABOSBT results. Results:. Nine thousand eight hundred ninety-two evaluations were completed using the ABOSBT for 449 different residents by 1,016 evaluators. 97.6% of all evaluations were scored level 4 or 5 (high levels of professional behavior) across all of the 5 domains. In total, 2.4% of all evaluations scored level 3 or below reflecting poorer performance. Of 428 residents, the ABOSBT identified 26 of 32 residents who were low performers (2 or more < level 3 scores in a domain) and who also scored “below expectations” by the PD at the start of the pilot project (81% sensitivity and 51% specificity), including 13 of these residents scoring poorly in all 5 domains. Evaluators found the ABOSBT was easy to use (96%) and that it was an effective tool to assess resident professional behavior (81%). Conclusions:. The ABOSBT was able to identify 2.4% low score evaluations (<level 3) for all residents. The tool was concordant with the PD for 81% of the residents considered low performers or “outliers” for professional behavior. The 5-domain construct makes it an effective actionable tool that can be used to help develop performance improvement plans for residents. Level of Evidence:. Level II