Benchmarking the Architect: A Methodological Framework for Evaluating Domain-Specific AI Agents in Software Design

Abstrak

This paper presents a methodological framework for evaluating domain-specific AI agents in software architecture design. We develop and benchmark a Retrieval-Augmented Generation (RAG) agent tailored for architectural decision-making, comparing it against a general-purpose LLM baseline. The evaluation employs four metrics: Decision Quality, Pattern Adherence, Response Time, and Output Consistency. A pilot study of 30 experimental runs across five representative scenarios demonstrates that the proposed agent achieves substantial improvements in decision quality (+66.4%, Cohen's d = 7.04) and pattern adherence (+75.7%, Cohen's d = 9.87), with significantly reduced output variability. These findings provide empirical evidence for the value of domain-specific AI agents in software engineering and establish a replicable benchmark for future research.

Identifier

• OpenAlex: https://openalex.org/W7166056859
• DOI: https://doi.org/10.5281/zenodo.20900246