THE ESSENTIALS IN 30 SECONDS

  • Catching JiTTests use large language models to generate tests for each code change that are meant to fail if the change introduces a bug.
  • Meta reports 74 serious privacy bugs caught at the diff stage in the first half of 2026, a 60% true-positive rate reported by engineers and 20% fewer privacy incidents.
  • In a 'human-on-the-loop' model, engineers set review criteria and check outcomes in aggregate rather than inspecting every generated test.

Meta published a post on its engineering blog on October 29 describing how it now runs "Catching JiTTests" at full scale. The system uses large language models to write tests on the fly for individual code changes, with the aim of catching bugs before they are merged into Meta's codebase. The post builds on a research paper released in January and an earlier blog post from February.

How just-in-time catching tests work

Conventional test suites are written by people, live in the codebase and must be maintained. Meta's argument, set out in February by Meta's Mark Harman, is that AI coding agents now change code faster than such suites can keep up. JiTTests take another route: when a code change (a "diff" in Meta's terms) arrives, the system infers what it is meant to do, creates deliberately faulty variants of the code, and generates tests designed to fail if the change has introduced a regression. The tests are then discarded, so nobody has to maintain or review test code.

According to a summary of the new post, Meta runs two workflows. A "dodgy diff" workflow treats every change as a potential bug, while an "intent-aware" workflow reads the change's title and summary to infer its purpose and likely risks. Candidate failures are filtered by an ensemble of LLM judges and a rule-based system called RubFake before anyone sees them.

The figures Meta reports

Meta says 589 engineers engaged with the system, 64% of those it contacted, and that engineers rated 60% of reported catches as true positives. In the first half of 2026, the system reportedly caught 74 serious privacy bugs at the diff stage, and privacy incidents of the kind Meta classes as SEVs fell by 20%. The company calls the approach "human-on-the-loop": engineers define review criteria and examine results in aggregate, and are pulled in only when a catch justifies it.

The January paper gives a sense of the earlier, smaller trials. Across 22,126 generated tests, change-aware methods produced four times more candidate catches than standard "hardening" tests, and automated assessors cut human review work by 70%. Of 41 candidate catches reported to engineers, eight were confirmed as real bugs, four of them serious.

Caveats

All figures come from Meta and have not been independently verified. The 60% true-positive rate also means that roughly four in ten reported catches were judged false alarms, and the post's full methodology could not be reviewed for this article.