Google and Mercor have opened a new test that asks AI agents to do real engineering work, and every leading model fails it. Physical-Engineering’s Last Exam, known as PLE, gives agents full project folders for mechanical, electrical, civil and structural jobs, then grades the finished design against safety codes. On the hidden test set, no frontier model scores 50%, with the best configuration reaching only 48.3%.
A test built like a real project
PLE contains 85 long-horizon tasks: 30 mechanical, 27 electrical, 18 civil and 10 structural. Instead of answering a textbook question, an agent must work inside a containerized project folder that mirrors what a practicing engineer inherits on the job, complete with drawings, spreadsheets, emails and earlier revisions that may no longer agree.
Each task holds an average of 61 files, or 5,170 across the benchmark, including PDFs, spreadsheets, emails, two-dimensional CAD drawings, three-dimensional BIM models and superseded revisions. The agent has no internet access and up to six hours to find the governing codes and physical limits, run calculations and produce a deliverable a licensed reviewer would approve.
Licensed engineers from Mercor’s expert network wrote 44 tasks end to end, building the project folder, the prompt and the grading rubric from real professional work. Google built the other 41 tasks with an adversarial loop, in which one agent drafts a task and another tries to break it, before engineers validated and hardened every task from both routes.
Google released 29 public tasks on November 3, together with grading tools and an evaluation container, and opened a competition on the remaining 56 hidden tasks. Thirteen Google-sponsored models can be run free on the competition servers, while prize details are still to be announced, so outside teams can test their own agent setups against the same hidden standard.
No frontier model clears half marks
Google tested 14 frontier models using seven publicly available agent harnesses, including Claude Code, Codex, Antigravity, OpenCode, OpenHands, Hermes and Pi. On the private test set, Claude Opus 5.5 in Claude Code leads at 48.3%, followed by GPT 6 Astra at 45.8% and Gemini 4 Argon at 40.6%, with every other tested configuration further behind.
The headline score hides a sharper failure. The top configuration satisfied 91% of individual constraints on co-authored tasks, yet only 41.3% of its deliverables passed every hard safety and feasibility check required for objective credit. In engineering, that distinction is the whole test: a design that meets most constraints but fails one hard check cannot be built.
That pattern matches what other demanding agent tests have found, where a capable model can look strong on parts of a job and still fail the job as a whole. The site’s earlier report on another demanding agent test put the best score at just 12%, a reminder that long, multi-step work exposes weaknesses that short question-and-answer benchmarks rarely catch.
Choice of model mattered far more than choice of harness. Scores spanned up to 50 points across models, while the same model moved by a median of only about 3 points across harnesses, although harnesses did affect run cost. The best configuration also took 27 million tokens and 63 minutes per task on average.
Why It Matters
Engineering output carries consequences that office work does not. A consultant’s deck is judged in a meeting, while an engineer’s design is stamped, built and expected to carry load for decades. It must obey physics and building codes, and a licensed reviewer must sign it, which is why PLE rewards cost efficiency only after every hard safety check is cleared.
For companies weighing agent deployment, PLE warns against handing autonomous design work to models on the strength of academic scores. Frontier models score above 90% on many academic benchmarks, according to Google, yet none reaches half marks when the job includes messy files, conflicting revisions and safety gates. Human review is the control between a draft and a structure.
Our read
PLE matters less as a leaderboard than as a change in what counts as evidence. Humanity’s Last Exam tested general knowledge, and Agents’ Last Exam tested digital work, but neither asked whether a model could produce something safe enough to build. By grading deliverables against codes, PLE moves the argument from what models know to what they can deliver.
Keep one number straight. The 48.3% figure here is the November 3 result on the 56-task hidden test set, as stated in both announcements. The research paper also reports a higher average across all 85 tasks, and the two figures are not interchangeable. The hidden-set figure is stricter, and it is the set competitors will be judged on.
The open competition is where PLE will prove itself or stall. If outside teams beat the baseline configurations, the benchmark will have done its job as a shared yardstick. If entries cluster near the same ceiling, the lesson is blunter: physical engineering is a discipline where reliability, not fluency, is the bottleneck.
Background
PLE follows a familiar pattern in AI evaluation, where each new exam is built because the last one stopped separating the best models from the rest. The paper behind PLE was first submitted in October and revised on October 30, but the November 3 release of public tasks, grading tools and the competition is the new development.
0 Comments