ArchBench: a good frame, a weak leaderboard (Category Architecture)
I read a short paper.ArchBench: Benchmarking Generative-AI for Software Architecture Tasks" The idea is sound: to assemble disparate evals for software architecture into one extensible system. But 19 July 2026 This is more a frame of the future benchmark than a working leaderboard for choosing a model. (You can evaluate the leaderboard for yourself. bench).
Team. SERC From India's IIIT Hyderabad, they assembled an open system from the CLI and React site. Each task is a plugin with its own data loader, prompt, parser and metrics. The general conveyor goes through the stages of loading, inferencing and estimating, saving the prompts, raw responses, use of tokens and latency. Results and new tasks are expected to be taken through PR.
But the bench itself is weak - everything. 5 task
- ADR generation (architecture decision records) Generation of serverless components Creating dynamic IoT services
- Establishment of microservices
- Restoration of traceability These tasks do not cover architectural analysis, dependency thinking, large-scale refactoring, or dealing with compromises. Moreover, end-to-end only ADR writing and traceability restoration are automated in CLI, and three more tables with metrics are transferred from the initial studies, their piplins for evaluation are still integrated.
Metrics mix things up ROUGE and BERTScore measure similarity of text (ADR-writing) Test pass rate - code functionality when generating services and functions This is far from assessing the quality of architecture. Agent sandbox-environments for the use of tools the guys do not yet.
The set of models is also surprising: GPT-3.5old GPT--4, Flan-T5/T0, CodeQwen and DeepSeek of previous generations.
In the generation of microservices there is Codex and Claude CodeBut without exact versions and configurations, which means comparing such figures is difficult.
The authors note that they expect community growth (here GitHub benchIf you'd like to cut it.)but in public history on the 19 No external contributions with new tasks or model results are visible in July. After the article, the interface and code were changed, but not the measuring body itself.
Overall, this benchmark looks like a concept, but it will become a real benchmark after the expansion of tasks, complete automation, the launch of modern versioned models and the emergence of independent participants. In the meantime, it's an invitation to build a bench.
#Architecture #AI #AI4SDLC #Evals #Research