Google and Mercor's Physical-Engineering's Last Exam finds no AI agent passes half of real design tasks

Google and Mercor have released Physical-Engineering's Last Exam (PLE), a benchmark that tests whether AI agents can carry out complete engineering assignments rather than answer textbook questions. Announced on the Google Cloud blog on November 3, it asks agents to deliver designs that a licensed engineer would sign off as safe, compliant with building codes and reasonably cheap. No configuration tested so far reaches 50%.

How the benchmark works

PLE contains 85 long-horizon tasks: 30 in mechanical engineering, 27 in electrical, 18 in civil and 10 in structural. Practising engineers wrote 44 of them, including prompts and grading rubrics. The other 41 were started by engineers, assembled by an agent pipeline drawing on engineering standards, then checked and made harder by engineers.

Each agent works in an isolated container holding a project folder of about 61 files on average, from PDFs and spreadsheets to emails, CAD drawings and 3D building models. It has to find the relevant codes and physical limits, run the calculations and produce the deliverables, with no internet access and a six-hour limit per task. Points are awarded only if a design first clears every hard safety and feasibility check; scoring then weighs code compliance and cost. Automated scripts check numerical outputs, and an LLM judge grades open-ended work against an expert rubric, according to the project site.

What the results show

Google says it tested 14 frontier models across seven public harnesses, including Antigravity, Claude Code, Codex, OpenHands and OpenCode. On the public leaderboard, Claude Opus 5.5 running in Claude Code leads with 48.3%, followed by GPT-6 Astra in Codex at 45.8%, Claude Sonnet 5.5 at 43.5% and Google's own Gemini 4 Argon at 40.6%. Several open-weight models score under 7%.

The authors highlight a gap between partial and full correctness: the best configuration met 91% of individual constraints on co-authored tasks, yet only 41.3% of its deliverables passed all hard checks. They also report that results vary by up to 50 points between models but by a median of only three points between harnesses for the same model, although harnesses change the cost of a run noticeably.

Limits and access

Only 29 tasks, the grading tools and an evaluation container are published on GitHub; the remaining tasks form a private test set, so the leaderboard cannot be fully reproduced by outsiders. Teams can submit their own agent setups for the private set; prize details are still to come. Because part of the task set was assembled by an agent pipeline and open-ended work is graded by an LLM judge, the scores say more about how models rank against each other than about whether an agent could replace an engineer's sign-off.