Problems With SHAP and LIME in Interpretable AI for Education: A Comparative Study of Post-Hoc Explanations and Neural-Symbolic Rule Extraction

Abstract

Given that education is classified as a ‘high-risk’ domain under regulatory frameworks like the EU AI Act, ensuring accurate and trustworthy interpretability in educational AI applications is critical due to its profound impact on student learning and development. This study compares a knowledge-based artificial neural network (KBANN) with a conventional artificial neural network (ANN) in the context of educational predictive modeling, focusing on generalizability, interpretability, and the fidelity of post-hoc explanations. While both models demonstrate comparable predictive performance, KBANN uniquely integrates structured educational knowledge, aligning more closely with essential educational principles and causal relationships. Post-hoc explanation methods, such as Kernel SHAP, Permutation SHAP, and LIME, were applied to the ANN to interpret its decision-making process, revealing significant variability in the assessment of feature importance. Through simulations based on the extracted rules from KBANN, we further examined the fidelity and reliability of these methods. Our findings, from global feature importance and correlation analyses, showed that post-hoc methods often fail to accurately reflect the structured knowledge learned by the model, misattributing importance to less relevant features. This misalignment and the resultant discrepancies in feature interpretation raise concerns about the reliability of these explanations, suggesting they may not always provide a trustworthy or accurate basis for understanding the predictive models. Finally, KBANN’s rule extraction achieved significantly lower computation times compared to post-hoc methods, highlighting its practical efficiency. These findings collectively underscore the limitations of post-hoc explanation methods in conveying the true reasons behind model predictions, urging caution among stakeholders and researchers when using these methods for interpretability in AI for education.

Keywords

• Learner modelling
• personalized learning
• AI for education
• explainable AI
• neural-symbolic computing
• SHAP