Vals AI launches Science Index for research tasks
Vals AI released its Science Index on October 16, combining three existing benchmarks into a single comparison of AI research capabilities. The index gives equal weight to BioMysteryBench, MysteryMechanism and Terminal-Bench Science, covering biological data analysis, experimental investigation and research workflows across multiple scientific disciplines. Each component contributes one third of the final score, so a larger task collection does not receive extra weight.
The new product is the combined index. Its component tests predate the launch, including BioMysteryBench, which Anthropic introduced in April.
Astra leads the combined ranking
GPT-6 Astra leads the published index at 65.09%, followed by GPT-6.1 Sol at 59.63% and Claude Sonnet 5.5 at 59.12%. Vals lists the corresponding costs per test as $8.03, $1.36 and $13.24. Sol therefore sits close to Sonnet on the aggregate score while costing considerably less in these evaluations.
Within Terminal-Bench Science, rankings vary by scientific discipline. Vals says Astra leads life and mathematical sciences, Sonnet leads physical and engineering sciences, and Sol shares the lead in earth sciences. Astra also places seventh on the broader Vals Index, illustrating how much a model’s position can change with the work being measured.
What the three tests measure
BioMysteryBench asks agents to investigate real biological datasets with identifying metadata removed. Recovering that information can require inspecting files, writing analysis code and querying biological databases. Vals runs the evaluation in bioinformatics sandboxes with a limit of 200 agent turns and four hours per task. Its reported scores average three complete runs, with refusals and failed answers receiving no credit.
MysteryMechanism tests experimental reasoning through 222 hidden mathematical relationships. Agents receive anonymous inputs, physical bounds and two initial observations, then choose additional experiments within a fixed budget. They have a persistent shell but no internet access or information about the underlying scientific domain. The submitted mathematical expression is checked against private probes, rewarding a functionally correct relationship rather than a particular name or notation.
Terminal-Bench Science contains 70 research workflows contributed by practicing scientists. They span life, physical, mathematical, engineering and earth sciences. An agent might reconstruct a signal, calibrate a model or produce a mathematical result inside a sandboxed terminal. Each task’s own verifier grades the resulting artifact on a pass or fail basis, with no partial credit. Vals evaluates the full set from version 0.1.0.
How to read the results
The testing setup matters. Vals is moving its terminal evaluation from Terminus 2 to mini-swe-agent, and the leaderboard identifies the harness used for each model. Its figures come from one run per model, with retries allowed for infrastructure or provider errors. Vals warns against directly comparing them with the official benchmark leaderboard, which uses vendor agents and three trials per task.
Biology results carry another important qualification. Anthropic labels tasks as human difficult when its expert panel did not solve them. That does not establish that nobody could. The benchmark’s authors also acknowledge that validation can show a signal exists in the data without guaranteeing that a person or model can uncover it from scratch.
Taken together, these tests provide a focused comparison of how agents handle defined research problems. The aggregate score should be read within that scope. It does not establish how often a system will make an original scientific discovery. For researchers choosing a model, the component results and evaluation setup remain important alongside the headline ranking.
The broader question of how researchers should evaluate AI also underpins the Simons Foundation initiative to help mathematicians assess model outputs.
Illustrative archival photograph of Mileidys Perez working at Vall d’Hebron Institute of Research in Barcelona in 2016. Photo by Alessandro Ventura, shared as axventura on Wikimedia Commons under CC BY 4.0. The photograph is unrelated to Vals AI.
