Ai2 finds AI novelty judges can change verdicts when the framing changes
A research idea can receive a different novelty verdict without the idea changing. Ai2’s study of LLM judges shows why teams evaluating AI research assistants need to test the evaluator as carefully as the idea generator.
Ai2 announced the benchmark on October 27, 2026. The paper first appeared on October 1 and was revised on October 26. This is a research evaluation release, rather than a new model or a November launch.
The headline result depends on the question
On one Human+Generated comparison, GPT-5.4’s accuracy fell from 93.5% to 40.9% when a prompt stopped telling it that reviewers had found exactly one of the two ideas novel. That is a 52.6 percentage point change on the same idea pairs. It is not an overall scientific reasoning score.
The research paper distinguishes absolute judgments about one idea from comparisons between two. The former use macro-F1, which balances the novel and non-novel classes. The headline pairwise metric counts ties as failures. Its reference procedure combines six calls, three for each presentation order, so a verdict is not necessarily one model response.
The prompts also change substantive framing. Asking which idea reviewers judged novel supplies an authority cue that simply asking which idea is novel does not. The experiment measures sensitivity to plausible evaluation choices. It does not establish that every wording change has the same effect.
The generated ideas have weak labels
The released repository describes 154 higher-novelty human ideas and 145 lower-novelty human ideas, selected using ICLR 2026 reviews and rating extremes. A separate pool pairs the same 154 higher-novelty ideas with 154 generated ideas. Human+Generated therefore contains 308 individual items or 154 paired comparisons.
The generated pool is assumed to have lower novelty. Its default generator is Claude Sonnet 4.5 making a single proposal without literature access, tools or feedback. These are weak labels, not independent expert verdicts on every generated idea.
One expert checked 30 pairs, concentrated on difficult cases, and preferred the human idea in each. That is a limited sanity check, not representative validation of the entire dataset. The selected human papers are also far from a random sample of science. Neither comparison supports a universal claim that AI systems cannot produce original research.
Extra search does not settle the question
Ai2 reports that retrieval and larger reasoning budgets did not consistently improve judgments. In a separate comparison using one call per idea, an agentic GPT-5.6 Sol judge with iterative search reached 0.78 macro-F1 versus 0.89 for Claude Opus 4.6 with standard retrieval, at roughly ten times the cost. These are results for the tested configurations, not a rule that literature search is unhelpful.
The more durable finding is disagreement. If two setups reverse their ranking of unchanged ideas, the evaluation is unstable even when the correct label is uncertain. A high score from one prompt can conceal that fragility.
What researchers can use
The repository provides data, evaluation configurations and a pipeline for collecting new review-based examples. It offers individual and paired formats, along with abstract and normalized research-plan versions. The code uses Apache 2.0. The data uses ODC-BY 1.0 with additional attribution and source terms described in the data documentation. Running judges calls external APIs and carries usage costs.
For a practical audit, keep the candidate ideas fixed while varying the judge, prompt, presentation order and retrieval setup. Report disagreements and ties alongside aggregate scores. Check a sample with qualified reviewers, and make the sampling method clear. A pipeline used to discard proposals needs particular scrutiny: rejected ideas may disappear before a researcher sees them.
This is a different failure mode from the changing-fact tests in our StateBench coverage, but the evaluation lesson is related. A capability should be tested in the conditions where a workflow will rely on it. Success on a convenient proxy can leave the actual weak point unmeasured.
Featured image: AI generated editorial illustration using Ai2’s official wordmark. The viewing frame and papers are conceptual, not an Ai2 product or benchmark screenshot. Ai2 is not a sponsor of this article.
