Momentic publishes QA Bench, a 1,396-bug test of QA agents in which its own Mo leads but misses most bugs

Momentic, a software testing startup from Y Combinator's Winter 2024 batch, used a Launch HN post on November 2 to publish QA Bench, a benchmark for AI agents that look for bugs by using an application the way a tester would. Its own agent, Mo, which the company first unveiled in late September, comes out on top. Momentic stresses that it designed the test, ran it and graded it.

What the benchmark measures

QA Bench uses four open-source web apps, Gitea, Metabase, Twenty and Saleor, split into 20 product areas containing 1,396 labeled bugs. According to the methodology post by chief executive Wei-Wei Wu, candidate bugs came from bug bashes run by Mo, Claude Code and Codex; each was reproduced, cross-checked by the other agents and then accepted or rejected by a human reviewer. Agents get one area in a fresh sandbox with identical seeded data, no access to source code and a 30- or 60-minute budget. An LLM judge, GPT-6.1 Sol, matches their reports to the labels. Bugs are weighted by severity, and false reports count against the score.

The results, and how to read them

In the published 60-minute averages, Mo matched about 250 bugs per pass, for 17.9% recall and 84.7% precision at $2.88 per matched bug. The strongest Codex setup (GPT-6.1 Sol with Playwright MCP) matched 94 bugs at 6.8% recall, with higher precision (89%) and a much lower cost of $0.56 per bug. The best Claude Code configuration, using Opus 5.5, reached 2.8% recall, and Devin 1.9%. The Launch HN post gives slightly different figures for Mo, 269 bugs, 19.3% recall, 84% precision and $2.67 per bug, apparently from a different aggregation; the averaged tables on the blog and GitHub are the more conservative reference.

Reading the results as Mo catching 84% of bugs would be a mistake: that figure describes how often its reports are valid, not how many bugs it finds.

Caveats acknowledged by Momentic

The company lists several limits. Labels partly came from three of the graded agents, which may share blind spots. Judging errors feed directly into scores. Gitea accounts for 11 of the 20 areas, there are only three runs per area with no error bars, and time budgets do not cap spending: a Mo pass costs around $720, against about $53 for the leading Codex setup. Hacker News commenters asked for a human QA baseline and questioned whether a custom harness will hold its edge as frontier models improve. The dataset and harness are public, so outside teams can try to reproduce the results.