Comparative Evaluation of BM25–FAISS and Small-LLM–GPT in Retrieval-Augmented Generation Concept Map Assessment

Abstrak

The development of Large Language Models (LLMs) has opened up new opportunities in the development of automated concept map-based assessment systems. One promising approach is Retrieval-Augmented Generation (RAG), which combines search capabilities to find relevant information with generation to produce more meaningful context-based assessments. This study compares two search methods, namely BM25 based on keyword matching and FAISS based on vector representation, as well as two generative models, namely Small-LLM and GPT, in the task of concept map proposition assessment in the relational database domain. The results show that the FAISS–GPT combination provides the best performance with a Macro-F1 score of 0.338, a QWK score of 0.146, and the lowest error with an MAE of 0.973 and an RMSE of 1.321, indicating a slight but noticeable improvement in agreement with expert scores compared to other configurations. Additionally, this combination also displayed an Explanation Relevance Score (ERS) of 0.79, demonstrating GPT's ability to generate more relevant, consistent, and human-like explanations. In contrast, Small-LLM had lower accuracy despite excelling in computational time efficiency, making it a viable option for resource-constrained systems. Overall, the results of this study confirm that the integration of dense retrieval FAISS and large GPT language models can improve the quality of concept-based automatic assessment in terms of accuracy, consistency, and semantic relevance, thereby potentially strengthening concept-based learning systems in digital education.

Identifier

• DOI: https://doi.org/10.22219/kinetik.v11i1.2594
• OpenAlex: https://openalex.org/W7131350552