StateBench shows why long context does not guarantee reliable state tracking
Ai2’s StateBench asks a question that context window sizes cannot answer on their own: can a model keep track of what is true now after thousands of changes? The benchmark separates looking up a fact from repeatedly updating it. Its results suggest that fitting a long history into a prompt does not guarantee reliable state tracking.
Ai2 introduced StateBench on November 3, following a paper posted on November 2, 2026. This is a research benchmark and public evaluation toolkit, not a new assistant feature or a test of persistent memory between conversations.
Four ways a fact can change
The StateBench repository describes four levels of difficulty. Immutable assignments require recalling a value that never changes. Reassignment replaces an old value, so finding the latest assignment can suffice. Self referential updates require carrying a result forward, as with a running total. The hardest level adds dependencies between variables, including copying values and swapping them.
Consider a tiny ledger. Setting a balance to five and later replacing it with eight is different from adding three to whatever the balance currently contains. Copying another account’s current balance adds another dependency. These examples illustrate why retrieving the most recent matching sentence is not always enough.
The benchmark’s matched tasks keep variables, values and update positions aligned while changing the update operation. That makes the comparison more informative than simply making every harder test longer or filling it with different material.
Where the models struggle
The frontier tests covered Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. Ai2 reports that tested frontier models reached 13% accuracy or less on the longest problems in the hardest tier. That figure is specific to those problems, not an overall accuracy score. The research paper reports failures on difficult tasks at input lengths between 10,000 and 64,000 tokens. Models had to generate the final state rather than choose among five answers.
A separate controlled experiment trained six small, 275 million parameter architectures on sequences of 8 to 256 assignments. Recurrent and hybrid designs generalized farther on simpler tasks, with testing extending to 64,000 assignments. Assignments are not tokens, and that 250 times training length range should not be mistaken for a result on the hardest tier. The flexible Gated DeltaNet variant generalized to roughly ten times the training length on that tier.
These are synthetic tasks with a fixed set of variables. They do not cover every challenge in an evolving conversation, such as introducing new entities. Input format also affects performance. The results therefore support a targeted evaluation lesson rather than a universal ranking of commercial model families.
What developers can use
The public StateBench dataset includes canonical traces, alternate representations, final states and metadata for reproducing evaluations. The repository supplies generation and evaluation tools. Its code is licensed under Apache 2.0. The Hugging Face dataset card does not specify a separate dataset license.
For teams building assistants, a useful next step is to test changing facts separately from document retrieval. A workflow can correctly locate an old instruction yet still apply the wrong current value. Evaluate corrections, replacements and dependent updates, then inspect which kinds of change cause errors. A larger prompt budget is helpful only if the system uses the added history correctly.
Keeping information across sessions raises a separate set of questions. Our coverage of Anthropic’s agent memory research examines stored notes, retrieval and review controls. StateBench instead isolates how models follow updates within a supplied sequence.
Featured image: AI generated editorial illustration using Ai2’s official wordmark. It depicts a conceptual state tracking mechanism, not an Ai2 product or interface. Ai2 is not a sponsor of this article.
