Transformer Encoder Model for Sequential Prediction of Student Performance Based on Their Log Activities

Abstract

Learning management systems (LMSs) have been used massively due to the growing utilization of distance learning. This advancement has led to increased educational data that can be analyzed to improve the quality of the learning process. Learning analytics (LA) is one of the most important methods that can be used to analyze student performance. In this paper, we proposed an LA method based on deep learning, i.e., transformer encoder, to sequentially predict the student’s final performance based on log activities provided by an LMS. The objective is to predict at-risk students of failing so that they can be mitigated as soon as possible. The proposed model was evaluated on the Open University LA Dataset (OULAD) for daily or weekly prediction. The results show that the model could predict at the early stage with an accuracy of 83.17% on withdrawn versus pass-distinction classes. Meanwhile, for the other tasks, i.e., withdrawn-fail versus pass-distinction and fail versus pass-distinction tasks, the accuracy was at least 76% at the early stage. The proposed model was compared to the LSTM model. We found that the transformer encoder performed better than the LSTM, with the average difference values from 1% to 3% in terms of accuracy and from 3% to 7% in terms of F1-score for all tasks, based on the statistical testing. Furthermore, the ablation study using positional encoding, different feature aggregation methods, and weighted loss function for the imbalanced class problem was conducted. In OULAD, we found that model without positional encoding was better in all cases. Furthermore, the weekly feature aggregation and the use of a weighted loss function performed better in some cases.

Keywords

• Learning analytics
• transformer encoder
• student at-risk prediction
• massive open online courses
• sequential model
• imbalanced dataset