MotionGraphicsBench tests whether AI follows animation instructions
MotionGraphicsBench asks a narrower question than whether AI can make an impressive animation: can it move an existing design exactly as requested? Researchers from Figma and Stanford introduced the benchmark in a November 10 research announcement, following a paper submitted on November 7. It is an evaluation release, not a new animation feature inside Figma.
The benchmark supplies a complete static SVG and motion instructions for each of 100 tasks. That separates animating a design from inventing its visual elements. For teams comparing coding models, the useful output is not just a ranking: it is a record of which requirements each animation missed.
What the reported scores actually measure
The authors tested 32 models across 47 configurations. In their highest available reasoning settings, only four models exceeded 50% pass@1. GPT-6 Astra at maximum reasoning reached 67.7%; GLM 5.3 at maximum reasoning reached 39.8%. These are researcher-reported results, not ByteForward tests.
Pass@1 measures the success rate of one independent attempt, averaging tasks equally. The separate pass@32 experiment covers 15 models and asks whether any of 32 independent attempts succeeds. It does not measure a model repairing its animation after receiving feedback. A high repeated-attempt score should therefore not be read as first-try reliability.
Checks that explain a near miss
The paper evaluates rendered SVG frames using task-specific programs rather than requiring an exact visual match to one reference. An attempt succeeds only when every check passes. The controlled model comparison uses JavaScript animations written with GSAP.
The public Scissor example makes that distinction concrete. Its prompt requires two rigid halves to rotate around a shared screw, briefly close, then return to their starting pose. A human-rejected Kimi K3 example passes seven checks but fails the nearly closed overlap check. The supplied report identifies the failure instead of reducing the result to an unexplained score.
For a developer, that is a more useful debugging starting point than “the animation looks wrong.” It still takes judgment to decide whether the benchmark’s requirements resemble a real project, and whether correcting one motion would disturb another.
Deterministic does not mean infallible
The researchers report 93.3% agreement with human judgments on a balanced held-out set of 180 animations, versus 83.3% for a GPT-6 Astra vision-language judge. Their programs still made 12 errors: nine false rejections and three false approvals. That is evidence of stronger agreement in this experiment, not proof of universal correctness.
The paper also limits its scope. These 100 tasks do not cover large scenes, complex deformation or highly interactive behavior. Detailed prompts differ from a designer gradually clarifying a short request over several turns. Results with GSAP need not transfer unchanged to other animation systems.
Access, licenses and a safe starting point
The Hugging Face repository provides benchmark inputs, verification programs and 53,180 recorded model outputs, including the human-review collection. Its quick start documents Python, Node dependencies and Chromium rendering. Developers can inspect saved programs or verify compatible SVG frames without generating a fresh model response.
There are separate reuse rules. Datasets and benchmark media use CC BY-NC-SA 4.0, while repository-authored code, including verification programs, uses MIT. Original animation rights and dependency terms remain relevant. Public availability is not a blanket commercial license for the animation assets.
The README also warns that Python verification subprocesses are not a security sandbox. Its guidance is to use an isolated container or virtual machine without secrets for untrusted code or dataset files. Reports can contain complete model responses, so prompts and request metadata should not contain secrets.
Start by reading an existing task and its report before committing to a model comparison. Keep the generation settings and attempt budget alongside any score. Unlike Figma’s Code layers and Specs workflow, this release is research infrastructure: its value is making motion requirements inspectable, not promising that every design can now animate itself.
AI-generated editorial illustration by ByteForward. Figma logo referenced from official brand assets for identification. Not a benchmark example, product interface or endorsement.
