A new benchmark built from real requests shows the best AI coding agent fully finishes only about one task in four. Researchers from Alibaba and several Asian universities posted Co-Design Bench on 26 October, and it tests ten leading models on 273 real coding jobs drawn from more than 26,000 user sessions. OpenAI’s GPT-6 Astra came first, meeting 69.9 percent of each task’s requirements on average, yet it produced a fully correct result in only 26.4 percent of cases. The rest left at least one detail unfinished.
The test
Co-Design Bench starts from real work, not synthetic exercises. The team sifted 37,463 user requests from 26,314 coding sessions and kept the hard ones, where users rephrased a request or rolled back a change. The result is 273 tasks with 2,331 hidden acceptance criteria. Agents must edit a live web app, not build a demo from scratch.
The test runs on two tracks. Multimodal Editing gives agents 100 jobs that start from screenshots, images, audio, video or documents, such as pulling a figure from a receipt and updating every dependent total. Repository Editing gives them 173 larger overhauls of whole codebases.
Grading is deliberately strict, because the point is to catch partial work. One AI judge checks the finished app in a browser against the hidden criteria, and a second judge re-checks every pass, hunting for mock-ups or hardcoded answers. Points go only to features that actually work in the running app.
The results
GPT-6 Astra led the field, but the lead is thinner than it looks. It met 69.9 percent of each task’s requirements on average, yet fully satisfied only 26.4 percent of all tasks. Claude Opus 5 came second and Qwen3.8-Max third, while Grok 4.6 came last. The gap between doing most of a job and finishing it is the story here.
The split between the two tracks is stark. GPT-6 Astra fully solved 41 percent of the multimodal jobs but just under 18 percent of the repository overhauls, where one request can touch layout, design, interaction and content all at once.
The surprise is Qwen3.8-27B, a far smaller model, which climbed to second place on the larger overhauls. The authors warn against reading a general advantage into that result, since rankings shift with the setting. Size alone, in other words, does not decide who finishes the job.
Why the old tests stopped working
Older benchmarks made the same models look almost perfect. On WebCompass-edit, a test of web-app edits, agentic models score above 99 percent, because they can run, check and fix their own code in a closed loop. Co-Design Bench leaves a gap of more than 35 percentage points against those scores.
The authors argue the old tests measure how well an agent polishes its own work, not how well it handles a real user’s messy request. That distinction matters, because the request is what the developer actually sends, and the polish is what the agent adds for itself.
The harness matters as much as the model
The environment moved the scores too. Run in different harnesses, Qwen3.8-27B scored 75 percent on larger overhauls in one setup and 56 percent in another, and which model came out ahead changed with the setup. More compute did not fix it: GLM-5.2 burned nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored worse.
The limits
The benchmark has limits, stated plainly in the paper. Each task is a fixed request, so follow-up questions and mid-task corrections are not tested, and the AI judges grading the apps can make mistakes. The paper is an unreviewed preprint, and most of its authors work at Alibaba, whose Qwen3.8-Max finished third. The top model was not Alibaba’s.
Our Read
The headline number flatters the tools and indicts them at the same time. Meeting 70 percent of the requirements sounds close to done, and for a first draft it is. But software ships whole or not at all, and a change that leaves one detail wrong still needs a human to find it, fix it and re-test it.
That hidden cleanup is the work the demos never show, and it is where most of the time goes. The benchmark also quietly shifts the buying question: vendors sell the model, but the harness, the prompts and the checks around it decide what actually lands. A team that tests only the model is testing only half the product.
Why It Matters
For teams letting agents write code, the finding resets expectations. A tool that completes most of a change still leaves a human in the loop on nearly every task, so review time, not writing time, is the bottleneck to plan around. Buyers should test agents on their requests, in their own harness, before trusting a leaderboard score.
Background
Coding agents have moved fast from autocomplete to full change requests, and benchmarks have struggled to keep up. Earlier tests measured small, self-contained tasks, where agents now score near perfection. Co-Design Bench is the latest attempt to close that gap, following METR’s finding that real-world acceptance of AI-written code runs about 24 points below benchmark scores.
Source: The Decoder report and the paper
0 Comments