Publikation
Retrieval-Augmented Chatbots for Scalable Educational Support in Higher Education
Hassan Soliman; Hitesh Kotte; Milos Kravcik; Norbert Pengel; Nghia Duong-Trung
In: Lixiang Yan; Andy Nguyen; Ryan Baker; Mutlu Cukurova; Dragan Gasevic; Kaixun Yang; Yueqiao Jin; Linxuan Zhao; Yuheng Li (Hrsg.). Proceedings of the Second International Workshop on Generative AI for Learning Analytics co-located with the 15th International Conference on Learning Analytics and Knowledge (LAK 2025). International Workshop on Generative AI for Learning Analytics (GenAI-LA-2025), March 3, Dublin, Ireland, Pages 22-31, CEUR Workshop Proceedings (CEUR), Vol. 3994, CEUR-WS.org, 7/2025.
Zusammenfassung
Students of educational sciences participate in learning activities, where appropriate support and timely feedback
are crucial. However, providing scalable, personalized, and timely support becomes a major challenge. This work
focuses on developing a didactic chatbot based on a Large Language Model (LLM) and enhancing its potential with
existing learning materials. Retrieval Augmented Generation (RAG) allows the system to provide comprehensive,
context-aware answers to specific course questions. Previous results suggested that it is possible to distinguish
between different contexts in which students work and provide them with prompt responses that consider the
relevant material. This paper presents insights from the technical implementation and the first results on the
quality of LLM-based chatbot responses to content and organizational questions in an educational science module
for student teachers. We compare previous automated evaluations using GPT-4 with newly conducted human
evaluations of chatbot-generated results. Our experimentation demonstrated that the chatbot could achieve
the highest correct response rate of 87%. Furthermore, human evaluations conducted by five expert annotators
assessed the chatbot’s responses. The agreement between the majority vote of these human judges and the GPT-4
evaluation showed substantial alignment. This study helps to demonstrate the potential of generative AI in the
delivery of digitally supported courses.