Memento 3 tests game world rulebooks

Memento 3 describes an agent that learns game rules while its underlying language model stays fixed. The preprint, listed publicly on October 9, pairs Claude Opus 5 with a revisable written rulebook and an executable simulation.

The authors report completing all 25 public ARC AGI 3 games in 7,518 actions, about 44% of the human baseline. Its 100.0 Relative Human Action Efficiency (RHAE) score reaches the metric ceiling. The closest reported comparator also completes every game.

Proposed code must match the rulebook according to an LLM check and reproduce recorded observations exactly before planning uses it. A mismatch prompts revision. This checks observed transitions, leaving unseen behavior uncertain. Git history keeps earlier hypotheses and implementations available for inspection, comparison and recovery after an unhelpful change.

In three games, the rulebook variant used 9% fewer actions and 18% fewer agent turns than a version without it. The model weights remain unchanged. These are reported research results, without independent replication established here.

What the score measures

The benchmark presents unfamiliar interactive environments without explaining their goals. Agents must discover what actions do and adapt as they progress. The ARC Prize Foundation’s technical report separates the 25 public demonstrations from harder private environments intended to test generalization. The public games deliberately expose only part of the private set’s mechanics.

That distinction matters when interpreting a perfect public score. It establishes performance on that demonstration set under the reported setup. A result on unseen private games would answer a different question about transfer to unfamiliar mechanics. Readers should check which set a headline describes before comparing it with an official evaluation.

Action efficiency also counts interactions with the game rather than all computation. Internal reasoning and tool operations that leave the environment unchanged do not add actions. A system can therefore spend considerable effort simulating choices while using few moves. Token consumption, elapsed time and monetary cost need their own accounting before calling an approach cheaper.

A similar distinction appears in ByteForward’s TestPrism reporting, where checking generated tests against several correct implementations changes what a passing result establishes.

A demanding comparison already exists

Earlier research by Sergey Rodionov, revised in August, tested textual and executable world models with different reasoning settings. Its followup using GPT 5.6 Sol found that a textual variant could complete every public level at the highest tested effort while using 41% fewer actions than the human baseline. Requiring executable models was therefore unnecessary for that particular completion result.

The verification variant still had measurable advantages in those experiments. It scored higher than the textual version at both compared reasoning settings and completed the set at a lower setting. It also used substantially more of the study’s token cost proxy. A stronger score and lower overall operating cost did not automatically travel together.

These comparisons explain why a useful assessment should hold the underlying model, reasoning effort and software setup constant while changing the proposed mechanism. Single runs also leave uncertainty about repeatability. The older study explicitly limits its conclusions to public games and warns that later models could have encountered information about them.

Code can make assumptions testable

The broader approach has an established research history. WorldCoder, published at NeurIPS in 2024, used a language model to write Python representations of an environment and a planner to choose actions. Experience that contradicted the program triggered further debugging. Its studies included grid environments and task planning.

That architecture makes a practical distinction between generating a plausible explanation and testing what it predicts. Researchers can inspect and change a program’s transition rules, then reuse them for different goals. But the WorldCoder paper assumes deterministic environments and identifies exploration and program search as difficult parts of the problem. Successful game simulations therefore leave substantial questions about noisier settings.