Developing a Qualitative Evaluation Framework for Explainable Decision Support Systems from a User Experience Perspective

Abstrak

The growing adoption of Explainable Decision Support Systems (EDSS) across various decision-making domains has created a need for evaluation mechanisms that assess not only system performance but also the quality of user experience in understanding and trusting system recommendations. This study aims to develop a qualitative evaluation framework for EDSS from a user experience perspective by identifying the key factors influencing users’ perceptions of transparency, understandability, trust, and system usefulness. A qualitative exploratory approach was employed in this research. Data were collected through in-depth interviews and focus group discussions involving 25 participants consisting of system users, developers, and user experience experts. The collected data were analyzed using thematic analysis to identify the principal themes relevant to EDSS evaluation. The findings reveal that explanation transparency, information relevance, ease of interpretation, and user control are the primary dimensions affecting user experience and system acceptance. Based on these findings, a structured qualitative evaluation framework was developed to assess the quality of explainability in decision support systems. The proposed framework contributes to the advancement of Explainable Artificial Intelligence research and provides practical guidance for developers in designing systems that are more transparent, understandable, and user-centered.

Identifier

• DOI: https://doi.org/10.59890/ijist.v4i7.29
• OpenAlex: https://openalex.org/W7172343868