Ai2's StateBench finds frontier models fall to 13% or less on the hardest long state-tracking tasks

The Allen Institute for AI (Ai2) published StateBench on November 3, a benchmark designed to measure how well language models keep track of information that changes over the course of a long document. The release includes a technical report by Jackson Petty, William Merrill and Ashish Sabharwal, a dataset on Hugging Face and code on GitHub, which Unite.AI reports is under the Apache-2.0 licence.

Four levels of difficulty

StateBench asks a model to work out the final state of a set of variables after a series of assignments. Ai2 groups these problems into four levels. In the simplest, "r-trivial", a value is set once and never changes. In the "aperiodic" level, values are overwritten and the model must find the latest one. The "soluble" level adds updates that depend on the previous value, such as adding five to a variable. The hardest, "general finite", uses assignment by reference, for example copying one variable into another or swapping two, which the authors link to a classic permutation problem known as S5. According to Unite.AI, sequences range from 8 to 64,000 assignments, and each has one correct final state and four wrong candidates.

How current models perform

Ai2 tested four proprietary systems: Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. All of them performed well on the two easier levels. On soluble and general finite problems, however, accuracy fell sharply as sequences grew longer, reaching 13% or less on the longest general finite cases. Open pretrained models, including the 7-billion-parameter Olmo 3 transformer and Olmo Hybrid, mostly solved the trivial level and stayed at chance on the hardest level.

What training from scratch shows

To separate the effect of architecture from that of training, the team trained six 275-million-parameter models on sequences of at most 256 assignments and tested them on up to 250 times that length. Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels, out to tens of thousands of assignments, while transformers degraded on longer inputs and never learned the general finite level. On that level, a Gated DeltaNet variant with a more flexible memory update stayed near perfect up to about ten times its training length. The authors conclude that the weakness of pretrained hybrid models stems from training rather than architecture.

Ai2 acknowledges a limit: the framework assumes a fixed set of variables, so it does not yet cover documents or conversations in which new entities appear over time.