Almost Nobody Is Using ChatGPT to Write Academic Science Papers (Yet)

Abstract

We assessed 19,000 scientific introductions to measure the level of undisclosed use of ChatGPT in scientific papers published in 2023 and early 2024. We applied a “stylistics” approach that has previously been shown to be effective at differentiating AI-generated text from human-written text in a variety of venues. Ten different MDPI journals were selected for this study, and the rate of use of undisclosed AI writing in these journals was fairly consistent across the journals. We estimate that ChatGPT was used for writing or significant editing in about 1 to 3% of the introductions tested. This analysis is the first systematic study of detecting undisclosed ChatGPT in published manuscripts in cases where obvious indicators, such as phrases like “regenerate response”, are not present. The work demonstrates that generative AI is not polluting mainstream journals to any appreciable extent and that the overwhelming majority of scientists remain hesitant to embrace this tool for late-stage writing and editing.

Keywords

• ChatGPT
• generative AI
• MDPI
• academic publication
• AI text detection
• XGBoost