Does ChatGPT Help Novice Programmers Write Better Code? Results From Static Code Analysis

Abstract

In the realm of AI-enhanced programming education, there is growing interest in using such tools to help students understand good coding principles. This study investigates the impact of ChatGPT on code quality among part-time undergraduate students in introductory Java programming courses, who lack prior Java experience. The source code of 16 students from the control group (without ChatGPT) and 22 students from the treatment group (with ChatGPT) who completed identical programming exercises focused on coding conventions was analyzed. Static code analysis tools assessed adherence to a common coding convention ruleset and calculated cyclomatic and cognitive complexity metrics. The comparative analysis shows that the ChatGPT-assisted group significantly improved code quality, with fewer rule violations and reduced cyclomatic and cognitive complexities. The treatment group adhered more closely to coding standards and produced less complex code. Violations primarily occurred in line length, final parameters, and the extensibility of object-oriented programming (OOP). These findings suggest that ChatGPT can be beneficial in programming education by helping students write cleaner, less complex code and adhere to coding conventions. However, the study’s limitations, such as the small sample size and novice status of participants, call for further research with larger, more diverse populations and different educational contexts.

Keywords

• Programming education
• ChatGPT large language models
• static code analysis