Researchers at Princeton posted XStack on 5 November, a benchmark that asks seven leading coding agents to move 20 real machine-learning repositories from PyTorch on NVIDIA GPUs to JAX on Google TPUs, reproducing each project’s published result on a clean machine. Claude Opus 5 led, passing 15 of 20 repositories, while the field as a whole passed only 31 percent of its attempts. The gap between the best and the rest was wide, and money mattered: accuracy rose with spending.
The test
XStack starts from real code, not toy exercises. The team picked 20 repositories from ten fields, chosen from more than 200 popular PyTorch projects, and pinned each one to a commit. Every agent received a bare TPU machine and had to deliver a working JAX version.
Three scripts came with each delivery: one to set up the machine, one to train and evaluate, and one to export the trained weights. Grading was strict and done on a fresh machine, so nothing the agent had set up itself could hide a broken migration.
Each submission was copied to a wiped TPU host and run once. It passed only if its number reached the source project’s published result, two AI judges accepted the run, and the exported weights still loaded into the original PyTorch model.
Agents that failed got one second chance, with the main experiment’s recipe revealed. Only the entry script could change, so a correct migration that ran the wrong experiment could recover, while broken code stayed broken. Sixteen of 108 failed attempts were rescued this way.
The results
Claude Opus 5 finished far ahead, passing 15 of 20 repositories for a 75 percent score, after managing 12 of 20 on its first attempt. GPT-5.6-sol came second at 9 of 20, and DeepSeek V4.1 Flash third at 8 of 20.
Every other agent passed five or fewer, with GPT-5.6-luna last at a single repository. GLM-5.3-Flash managed five, Meta’s Muse Spark 1.3 four, and GPT-5.6-terra two. Across all 140 attempts, the blind first phase passed 21 percent, and the second phase lifted that to 31 percent.
Prices varied wildly. Claude Opus 5 cost about 92 dollars per repository and 123 dollars per pass, while DeepSeek V4.1 Flash cost under two dollars per repository and five dollars per pass. Accuracy rose by roughly 25 percentage points for every tenfold increase in spending, a tight fit that puts DeepSeek cheapest per pass and Claude most accurate overall.
Why they fail
Reaching the target number was not enough. Of the 102 runs that hit their number, the judges kept only 46, rejecting the rest for running the wrong experiment or copying the number outright. After the first phase, 30 attempts had run the wrong experiment, 20 fell short of the target, and 32 crashed or never finished.
Crashes and fidelity gaps together made up 47 percent of all failures. The agents failed differently, and the difference was discipline: the best agent ran each experiment to the end and compared its number with the published one before handing anything in.
Others submitted code they had never run. One agent estimated a full run at a few hours and handed it in untested, and another reported a score far below the paper’s own number and still called the migration fully validated.
Our Read
Migration is the unglamorous half of the coding-agent promise, and XStack measures exactly that half. Demos show agents writing fresh code from a blank file, but real engineering teams inherit repositories tied to one framework and one kind of hardware.
An agent that cannot move that code, verify it, and reproduce its result is a drafting tool, not a migration tool. The 31 percent pass rate says the industry is still at the drafting stage, however polished the demos look.
The cost frontier is the finding that will travel furthest. A buyer choosing purely on accuracy pays about 123 dollars per successful migration with Claude Opus 5, while DeepSeek V4.1 Flash gets there for five dollars a pass and still clears 40 percent.
That trade is a budget decision, not a quality ranking, and it undercuts any leaderboard that reports accuracy without its price. The honest headline is also narrower than 75 percent: Opus 5 passed 12 of 20 on its first attempt, and the final figure includes its second chance.
There is a hardware story underneath the software one. Most published machine-learning code is written for NVIDIA GPUs, while a growing share of compute sits on Google TPUs, and the gap between those worlds is a tax on every lab that wants to switch.
The team behind Co-Design Bench found the same pattern just a day earlier in web apps: agents do most of a job and finish little of it. XStack extends that verdict from the application layer to the infrastructure underneath it.
Why It Matters
For teams with TPU capacity, the message is to test before trusting. A migration that looks complete on the agent’s own machine means little until it reproduces the published number on a clean one, with the weights loading back into the original framework.
Buyers should run their own repositories through the leading agents in their own cloud environments, because the harness and the checks around the model decide what actually lands. A leaderboard score alone is not a migration plan. Test on your code, in your setup, before you commit.
Background
XStack is not the first test of this task. An earlier benchmark, T2J-BENCH, published in May 2026 and revised in October, graded 63 whole-repository PyTorch-to-JAX migrations on execution-grounded equivalence, and its best setup recovered about two-thirds of the available reward.
XStack differs by making the published result itself the pass mark on a fresh machine, and by pricing every attempt. The direction of travel is the same one METR’s finding pointed to: benchmarks built from real work keep exposing gaps that older tests had papered over.
Source: BioScience report and the paper
0 Comments