Artificial Analysis Index v5 changes the test, not just the ranking

Artificial Analysis has rebuilt its Intelligence Index around nine evaluations, with 60% of the weighting now tied to private tests or held-out answers. Its October 27 launch article introduces version 5 alongside a separate Progress Index for tracking model capability over time. The distinction matters: a changed test is not a clean before-and-after measurement of the same model.

What changed inside Index v5

The v5 methodology gives knowledge work, coding, science and general capability equal 25% weights. New arrivals are FrontierCode 2.0, with 132 coding tasks, and Terminal-Bench-Science, with 70 scientific workflows. HLE-Diamond replaces Humanity’s Last Exam with 1,000 questions. AA-Omniscience moves to version 1.1, while CritPt uses a private 40-question hard subset with code execution.

AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 remain. GDPval-AA v2.1, SciCode and AA-LCR v1.1 leave. Where tasks are repeated, the methodology averages attempts rather than selecting the best run. The suite remains primarily English and text-based, even though some tasks include images. It is not an exhaustive measure of every language, modality or product experience.

Private tasks can make memorized benchmark material less useful, but privacy alone does not prove an evaluation is contamination-free. It also limits what an outside reader can reproduce from public task files. That makes the published harness, scoring rules and task mix important context, not footnotes to the ranking.

Our earlier FrontierCode 1.1 coverage explains the emphasis on repository-maintainer acceptance. The new version belongs to a revised test suite, so old and new scores need their version labels.

Effort and cost change the comparison

At launch, the leading rounded composite scores were Claude Opus 5.5 at 58, GPT-6 Astra at 57 and GPT-6.1 Sol at 55, all at maximum reasoning effort. Those numbers do not establish whether a one-point gap is statistically meaningful.

The contrast within Opus is especially useful: Artificial Analysis lists 35 at low effort and 58 at maximum effort, with benchmark task costs of $0.98 and $10.04 respectively. Both entries use its Default Fallback setting. A model name without its evaluated configuration leaves out a major part of this comparison.

The cost methodology combines token consumption and prices within each evaluation, divides by task count, then applies the Index weights. Input, cached and output tokens can carry different costs. This is a benchmark-weighted cost per attempted task, not a subscription price, a guaranteed bill for your job or a cost per successful completion.

For a team choosing a coding or research assistant, the useful question is therefore not simply which bar is longest. It is whether the additional effort produces enough acceptable work on that team’s tasks to justify its latency and expense. This chart is a shortlist aid, not a procurement verdict.

Use the Progress Index for the historical question

Artificial Analysis explicitly says Intelligence Index scores across versions are not intended for long-run comparison. Its new Progress Index instead uses item response theory to estimate task difficulty and model capability together from overlapping benchmark results. Older and newer models do not need to have completed identical evaluations to be placed on that continuous scale.

That is a different statistical construction, not a simple relabeling of v5 points. A lower score after a benchmark revision is not, by itself, evidence that the model became less capable. Conversely, a high current composite does not explain which particular tasks improved.

How to use the new results

Start with the capability category closest to the work, then retain the exact model, effort setting and harness in the comparison. Test a small representative set with a clear acceptance rule. Record failed attempts as well as successful ones, and measure the cost of getting an acceptable result rather than treating every generated answer as useful output.

Keep the benchmark edition alongside any saved score. That small discipline helps distinguish a benchmark revision from a model regression, and keeps a promising aggregate result separate from evidence that a model fits a specific workflow. ByteForward has not independently rerun these evaluations.