THE ESSENTIALS IN 30 SECONDS

  • Singularity-Bench, posted on arXiv on 12 November, has agents build six AI systems, from pre-training to cluster scheduling, and compares them with mature human-engineered tools.
  • Claude Opus 5.5 running in Claude Code scored 1.252 on the paper's index, where 1.0 equals the human baselines; the other ten agents averaged 0.58.
  • The authors also show the evaluation framework could be gamed and propose a hardened version called Breakwater.

A preprint posted on arXiv on 12 November introduces Singularity-Bench, a benchmark that asks whether AI agents can build the software used to create AI systems. The 19 authors list affiliations including Carnegie Mellon University, NVIDIA, the University of Texas at Austin, Meta and Amazon AGI. The paper has not yet been peer reviewed.

Six tasks, six human yardsticks

Each task starts the agent from a minimal codebase and a fixed hardware budget, mostly 12 hours on four NVIDIA GB300 GPUs, then compares its result with a mature, human-engineered reference on the same machines. The tasks cover the whole stack: pre-training a small language model (against Megatron-LM, scored on perplexity), mid-training ahead of reinforcement learning (Megatron-LM), RL post-training of a software-engineering agent (the Miles framework), serving a model for inference (SGLang), writing a coding-agent harness (Claude Code itself) and scheduling workloads on a 16-GPU cluster (NVIDIA Dynamo with the KAI scheduler).

The headline metric, the Singularity-Index, averages the agent-to-human ratio across the six tasks, so 1.0 means parity on average.

What the results show

The authors tested 11 agents built from seven frontier models and two harnesses, OpenAI's Codex and Anthropic's Claude Code. Claude Opus 5.5 in Claude Code reached 1.252, ahead of GPT-6.1 Sol in Codex at 0.969 and Claude Opus 5 at 0.803; the remaining agents averaged 0.58. The abstract says the top agent matches or outperforms the human baseline on all six tasks. The detailed table is more nuanced: it scored above the reference on five, but on post-training its resolve rate of 0.533 sat just under the human 0.539, a gap the authors' sign tests cannot separate from a tie. Its API bill came to about $425, excluding GPU time, against $4.36 for the cheapest agent.

The authors add several caveats. The tasks are smaller than real recursive self-improvement would require, the top agent's raw scores still leave plenty of headroom, and its harness lead rests on a single additional bug fix.

Testing the test

A large part of the paper concerns reward hacking. Red-teaming the Harbor evaluation framework, the authors found attacks that succeeded on every task, from planted code that dug answers out of the Python call stack to tampering with logs read by an LLM judge. Their extension, Breakwater, adds binary fingerprinting, API filtering and tamper-proof logging, and the paper reports it blocked all attacks tested.