Skip to content
back to the archive page
#Architecture

R2ABench: From Requirements to Architecture—or Why PlantUML Skills Do Not Make an LLM an Architect (Category #Architecture)

#Architecture #AI #AI4SDLC #Engineering #Research #Evals

R2ABench infographic showing the gap between LLM-identified components and architectural relationships

I read the latest version of the R2ABench paper. The authors ask a good question: can an LLM turn requirements into an architectural representation of a system, rather than merely draw a plausible-looking diagram? The short answer is that models identify components reasonably well, but the architecture starts to fall apart at the relationships, the rationale for those relationships, and the traceability of decisions back to requirements.

R2ABench contains 68 projects: 17 master's-level student projects with formalized requirements and 51 open-source GitHub projects. Each one includes:

The student projects already had SRS documents. For the GitHub projects, the researchers reconstructed them from READMEs, documentation, interfaces, entry points, and repository configuration. GPT-5.4 assembled the draft, Claude Opus 4.8 searched for contradictions and unsupported claims, and three people then reviewed the findings. Inter-reviewer agreement was substantial: Fleiss' κ = 0.763. This is therefore not purely a greenfield task of turning a raw PRD into a new architecture; it is partly a reconstruction of an already implemented system.

The experiment ran four models—GPT-5, Claude Sonnet 4.6, DeepSeek V3.2, and Qwen3-Coder 480B-A35B—both directly and through MetaGPT, OpenHands, and Mini-SWE. The result was 1,088 architectural representations, evaluated in three layers:

  1. Does the PlantUML render and parse into a graph?
  2. Do the components, relationships, layers, and topology match the reference view?
  3. Does the diagram cover the requirements and architecturally significant requirements (ASRs), can decisions be traced to evidence, is the decomposition reasonable, and is the result readable?

GPT-5.4-mini evaluated the final layer. To avoid simply trusting LLM-as-a-Judge, the authors compared its scores with independent annotation of 272 outputs: 72.77% of its scores were within one point of the human consensus, and mean absolute error was 0.53 on a five-point scale.

The central result is a sharp gap between the nouns and verbs of architecture: components versus the relationships among them. Reference-component coverage ranged from 0.669 to 0.849 across configurations, while Edge F1 never exceeded 0.176. Here Edge F1 measures how accurately the model reconstructed directed relationships among architectural components: calls, dependencies, and data flows. Suppose the reference contains: API → Order Service → Payment Adapter → Payment Gateway If the model omits the Payment Adapter relationship or invents Order Service → Redis, its score falls.

The authors separately inspected some of the missing relationships and found that 27.9% genuinely could not be inferred from the SRS. Even after excluding those cases, aggregate Edge F1 remained around 0.11. Put simply, the model often understands that a system should contain an API, a payment service, and a database, but fails to recover who calls whom, where data flows, and which requirement justifies a dependency. Of the structural errors, 39.1% were invented relationships and another 30.2% were invented components. At the semantic level, traceability and architecturally significant requirements were the most frequent losses.

Another useful result is that agent scaffolding produced no consistent advantage. Qwen3-Coder with OpenHands achieved the best Edge F1 and graph-edit score, but generated parseable PlantUML in only 54.4% of cases and trailed direct GPT-5 on every semantic dimension. The agent layer changed the error profile; it did not turn the model into a reliable architect.

I see two practical conclusions.

1️⃣ Treat an LLM-generated architecture diagram as a hypothesis, not a decision. Do not merely check for familiar boxes. Verify every important relationship: which requirement or ADR supports it, which quality attribute it implements, and what would break if it changed. 2️⃣ An eval for an AI architect cannot be reduced to valid PlantUML, graph similarity, or one judge-model score. Syntax, structure, ASR coverage, evidence grounding, and bidirectional traceability need separate checks. Otherwise, we measure the neatness of the drawing rather than the quality of architectural reasoning.

The study has important limitations: - Only 68 projects - A single reference view per system - More complete SRS documents than teams usually have at the start of a project - GPT-5.4-family models participate in data preparation, graph alignment, and evaluation, creating same-family bias - An almost certain chance that the GitHub projects appeared in model pretraining data

Even so, R2ABench is a useful engineering signal. An LLM can already assemble a first architectural draft quickly. The expensive work remains the same: proving the relationships, preserving the constraints, and explaining why the system is designed that way.

P.S. For anyone who wants to explore the analysis, the benchmark data is available here.

#Architecture #AI #AI4SDLC #Engineering #Research #Evals