Comparative Analysis of the Risk of Hadith Errors in Question-Answering Systems Based on Large Language Models

Abstrak

The use of Large Language Models (LLMs) in Question-Answering (QA) systems is becoming increasingly widespread due to their ability to generate contextual natural language responses. However, the application of LLM-based QA systems in high-risk domains, such as religious information services, poses serious challenges related to accuracy, reliability, and ethical responsibility. In the context of hadith information, errors generated by AI-based systems have the potential to cause misinformation and undermine user trust. This study aims to conduct a comparative analysis of the risk of errors generated by Large Language Model-based Question-Answering systems in answering hadith-related questions. Four LLM models, namely GPT, Gemini, Claude, and LLaMA, were evaluated using a controlled set of designed prompts. The model outputs were analyzed and classified into four error categories: fabricated hadith, incorrect attribution, hybrid hadith, and distortion of meaning. The frequency of each error type was then analyzed comparatively to identify error patterns and risk levels for each model. The results show significant differences in the number and types of errors among LLM models, reflecting different risk profiles in the implementation of hadith QA systems. These findings indicate that LLM-based question-answering systems still have limitations in ensuring the accuracy and reliability of religious information. This research is expected to contribute to the development of a risk evaluation framework and support the implementation of ethical principles and responsible AI governance in artificial intelligence-based information systems.

Identifier

• DOI: https://doi.org/10.51244/ijrsi.2025.12120025
• OpenAlex: https://openalex.org/W7117546627