Development of a Tool for Evaluating the Influence of Engineering Students’ Perception of Generative AI on University Courses Based on Personality, Perceived Roles in Design Teams, and Course Engagement

Abstract

This research investigates the possible influence of students’ perceptions of emerging AI technologies on university courses, focusing on their knowledge and perceived usefulness within engineering design. An evaluation tool implemented in a Microsoft Excel workbook was developed and tested to perform the process of data collection through well-known questionnaires, data analysis, and the generation of results, facilitating attention to class compositions and measuring AI awareness and perceived usefulness. The study considers traditional aspects such as roles within design teams and the psychological factors that may influence these roles, alongside contemporary topics like Large Language Models (LLMs). Questionnaires based on well-established theories were administered during courses on product innovation and representation, assessing both primary and secondary design roles. Primary roles focus on technical skills and knowledge, while secondary roles emphasize problem-solving approaches. The Big Five questionnaire was used to characterize students’ psychological profiles based on the main personality traits. Students’ perceptions of AI involvement and usefulness in engineering design were evaluated using questionnaires derived from the consolidated literature as well. Data were collected via Google forms from both in-class and off-line students. The first results of the workbook adoption highlight some relationships between personality traits, perceived roles in design teams, and AI knowledge and usefulness. These findings aim to help educators enhance course effectiveness and align courses with current AI advancements. The workbook is available to the readers to collect data and perform analyses in different countries, education disciplines, and as time goes by, in order to add the longitudinal point of view to the research.

Keywords

• artificial intelligence—AI
• Large Language Models—LLMs
• engineering design
• engineering education
• evaluation tools
• Gen Z