An Applied Statistics dataset for human vs AI-generated answer classification

Abstract

Due to the increasing popularity of Large Language Models (LLMs) like ChatGPT, students from various fields now commonly rely on AI-powered text generation tools to complete their assignments. This poses a challenge for course instructors who struggle to identify the authenticity of submitted work. Several AI detection tools for differentiating human-generated text from AI-generated text exist for domains like medical and coding, and available generic tools do not perform well on domain-specific tasks. Those AI detection tools depend on LLM, and to train the LLM, an instruction dataset is needed that helps the LLM to learn the differences between patterns of human-generated text and AI-generated text. To help with the creation of a tool for Applied Statistics, we have created a dataset containing 4231 question-and-answer combinations. To create the dataset, first, we collected 116 questions covering a wide range of topics from Applied Statistics selected by domain experts. Second, we created a framework to randomly distribute and collect answers to the questions from students. Third, we collected answers to fifty assigned questions from each of the 100 students participating in the work. Fourth, we generated an equal number of AI-generated answers using ChatGPT. The prepared dataset will be useful for creating AI-detector tools for the Applied Statistics domain as well as benchmarking AI-detector tools, and the proposed data preparation framework will be useful for collecting data for other domains.

Keywords

• LLM
• Assignment
• Transformer
• AI