Ai2’s StateBench: frontier models fall to ≤13% on the hardest long state-tracking level
Important
Primary
On November 3, 2026, the Allen Institute for AI published StateBench, a long-context state-tracking benchmark with four difficulty levels (r-trivial, aperiodic, soluble, general finite). Ai2 reports Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna stay strong on easier levels, then drop to 13% or lower accuracy on the longest general-finite problems.
At a glance
Figures from Ai2’s November 3, 2026 StateBench post on Hugging Face.
Date
November 3, 2026
Levels
4 (r-trivial, aperiodic, soluble, general finite)
Frontier hardest
≤13% on longest general-finite (Ai2)
Frontier models tested
Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna
Direct-train max train
256 assignments
Direct-train max test
64,000 assignments (~250×)
Hardest arch note
Flexible Gated DeltaNet ~near perfect to ~10× train length (Ai2)
How to read this claim
Measurement — StateBench: frontier models Ai2 tested fall to ≤13% on the longest general-finite state-tracking problems — Ai2’s figure; name the level,…
Bill — A score is not a price. If you will call a model next, list input and output per million tokens on the vendor page.
Ai2 released StateBench with four graded state-tracking levels and reported frontier models falling to ≤13% on the longest general-finite problems.
What it introduces
StateBench — a public long-context state-tracking benchmark with four difficulty levels, data, and code.
Who benefits Anyone buying or building on long-context models for code, narratives, or multi-step entity tracking, and teams comparing transformers to recurrent or hybrid stacks.
Try this
When a lab says the model “handles long documents,” ask whether the claim is r-trivial lookup, aperiodic latest-value, soluble running totals, or general-finite reference chains — and at what length. Prefer a named StateBench level over a raw context-token number.
What we do not know
Exact token lengths and per-model tables for the “13% or lower” frontier point are chart-based in the post rather than a single numeric table row.
Whether hosted frontier models were run with tools or only sequence reading is described as reading assignments and generating final variable values.
Coverage of newly appearing entities over a conversation is explicitly out of scope for the current framework.
Evidence
PrimaryStateBench post. Four difficulty levels, the ≤13% frontier figure on longest general-finite problems, and the direct-training 64,000-assignment / ~10× generalization notes are from Ai2’s November 3, 2026 StateBench post on Hugging Face.
Independent Scores and architecture comparisons are Ai2’s. Third-party reruns of the frontier chart are not established by this post alone.
Still unclear? · Start here · ranks, prices, unknownStateBench: frontier models Ai2 tested fall to ≤13% on the longest general-finite state-tracking problems — Ai2’s figure; name the level, not only the context length.
Ai2 published StateBench on November 3, 2026 (Hugging Face blog; tech report, dataset, and code linked). The suite builds four comparable problems from the same assignment skeleton so only the update type changes: r-trivial (immutable), aperiodic (latest value wins), soluble (self-referential updates), and general finite (assignment by reference / swaps). On pretrained open models trained with documented data and ≤64,000-token contexts, almost all stayed near chance on general finite. On frontier models — Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna — Ai2 says accuracy on soluble and general finite fell sharply with length, to 13% or lower on the longest general-finite problems. Separately, 275-million-parameter models trained from scratch on StateBench (up to 256 assignments) were tested out to 64,000 assignments (~250× training length): Gated DeltaNet and hybrids stayed near perfect on the three easier levels at those lengths; transformers never learned general finite, while a more flexible Gated DeltaNet update stayed near perfect out to roughly 10× the longest training length on the hardest level.
Why it matters
A long-context marketing claim can still hide a failure mode StateBench names. The useful habit is which update class and which length, scored by whom — not “context window size” alone.
Problem format mattered for small pretrained models: RULER-style r-trivial could look near ceiling while spreadsheet-edit framing of the same problem was near chance.
Ai2 releases data on Hugging Face (allenai/state-bench) and code on GitHub (allenai/state-bench); the tech report is at allenai.org/papers/statebench.
Open questions in the post include Gated DeltaNet’s exact hierarchy placement under limited memory updates and tracking newly introduced entities over time.