EdTec-QBuilder: A Semantic Retrieval Tool for Assembling Vocational Training Exams in German Language

System Demonstration Track @ NAACL 2024

The Exam Assembly Task

In education, retrieving and assembling test items from a previously validated and curated item database into comprehensive examination forms is a real challenge. While a significant focus is on the research and development of models for Automatic Item Generation (AIG), the crucial task of assembling examination forms still needs to be explored. This gap presents an opportunity for research and development, especially in creating effective and efficient methods for assisted and automatic exam assembly.

While collaborating with the bfz group, Germany's leading vocational training provider, we identified the difficulties of curating an item database and filtering relevant test items for building (or assembling) formative examination forms. To address this challenge, we formulated the exam assembling process as an information retrieval task. We implemented and deployed a proprietary search service and tool to assist vocational educators in assembling examination forms. This system facilitates acquiring or crowdsourcing ground-truth expert data from manual exam assembly processes. As a system demonstration, we released EdTec-QBuilder, the public fork of our search service and tool.

System Implementation

The below figure describes the architecture of the search service. We provide a JSON Web Token-based service to authenticate and serve client requests from the SERP.

After user authentication, the endpoint validates the users' credentials in the user's database. When a user submits a query, the query is encoded into a vector-dense representation to approximate nearest neighbors' search against an NMSLIB precomputed index. To optimize the output of the initial search, the tool's core search module re-ranks the top 100 results with a cross-encoder. Then, it returns a response in JSON format with the resulting test items and their corresponding attributes. The client receives the system's output, and the SERP maps the JSON into a suitable format for the visualization of results. The client then selects relevant items; in the background, the SERP logs the chosen items and sends this data to the web tool.

Search Strategy Selection and Analysis

In this initial phase, as no ground truth data was available and focusing on delivering a functional and practical tool for our commercial partners, we have strategically implemented eight foundational search and retrieval strategies. These strategies serve as initial baselines, carefully chosen to define the core of our test item search and retrieval module. It's important to note that this is an early approximation, designed to evolve and improve as we gain insights and feedback in our ongoing commitment to excellence and innovation.

To overcome the cold start problem, we evaluated and analyzed how the proposed methods performed while interacting with the top 25 popular pre-trained sentence similarity models by employing a synthetic TREC-style methodology. We found that a cross-encoder passage re-ranker performed the best for our task. For more details about our experimental setup, check our relevant publication.

Code

To foster research in information retrieval and assisted and automated exam assembling models, we released the EdTec-QBuilder code. The code is available under Creative Commons Licence.

Code

Dataset

The item database comprises 5,624 test items across 34 high-demand vocational skills in the German job market. We augmented our original item database via ChatGPT3.5 to increase our experiments' robustness and scalability and to share a public version of our dataset. Below, under a Creative Commons Licence, you can download the public fold of the item database, and the anonymized TREC-style search runs the strategies output.

Download Dataset Download Search Runs

Papers

• TBA

Search & Retrieval Performance Leaderboard

We evaluated our experiments with classic information retrieval metrics. Below, we display the performance leaderboard sorted by nDCG performance. Overall, we tested 176 different pre-trained sentence similarity models and strategy combinations.

Toggle Table