Android Bench adds three models. Finishing the app is still the test
Google updated Android Bench on October 26 with Claude Sonnet 5.5 in Claude Code, GPT 6.1 Sol in Codex and Gemini 4 Argon in Antigravity SDK. Results now display reasoning effort too. The update archives the earlier Sonnet 5 and GPT 6 Sol entries.
That extra configuration detail matters. A coding assistant is a model working through tools, prompts and an execution environment. Treating the displayed score as a property of the model alone hides part of the system a developer would actually use.
Three new entries, two different tradeoffs
This comparison isolates the three newly added configurations. It is not a top-three ranking of the entire leaderboard. The chart and table use the live Android Bench results checked on October 27.
| Configuration | Pass rate | Reported interval | Full-run cost |
|---|---|---|---|
| Sonnet 5.5 / claude-code / max | 40.0% | 24.0% to 56.0% | $1,574.90 |
| GPT 6.1 Sol / codex / max | 29.3% | 15.3% to 42.7% | $99.50 |
| Gemini 4 Argon / antigravity-sdk / high | 21.3% | 10.0% to 33.3% | $267.60 |
Cost means the average for a complete 30-task benchmark run, not one task or a subscription price.
The pass-rate intervals overlap. These point estimates do not establish a decisive pairwise winner. Nor does the cheaper run establish a lower delivery cost. A team still has to pay for reviewing, debugging and finishing whatever the agent leaves behind.
A 99% completion score can still mean zero passes
Sonnet’s Bitwarden navigation migration averaged 95% completion but passed none of five runs. Sol’s BlueWallet conversion reached 99% and also passed none. Both cleared Signal’s Java-to-Kotlin Glide migration in all five runs. The first two examples failed tests, according to the per-task results.
These are different kinds of work, not interchangeable coding puzzles. Replacing an image-loading module is a narrower assignment than converting an app between frameworks. An aggregate rank cannot tell a developer whether a particular migration is likely to end with a working change.
The gap between partial credit and a pass is especially relevant to app maintenance. An implementation can contain most of the requested pieces and still fail a condition that blocks release. Counting how much code was produced would miss that distinction completely.
What Android Bench calls a finished task
The methodology uses 30 long-horizon tasks with five independent runs per task. Agents operate in isolated environments with Android virtual devices. Passing requires functional tests, visual requirements and constraints all to be satisfied. Completion is a separate partial-progress measure.
Checks include instrumented interactions, regression tests and evidence such as database state or outbound events. Visual assessment also uses Gemini 3.5 Flash to inspect screenshots and accessibility information. The long-horizon dataset remains private.
Those details set useful boundaries on the results. Model-based visual judging is not the same thing as a human accessibility review. And a private task set limits what an outside team can reproduce independently. Read this as evidence from a defined evaluation, not a guarantee that generated apps are ready to ship.
Use the task breakdown before choosing an agent
For a developer facing a similar migration, the specific task history is a more useful starting point than the overall order. Match the framework change and acceptance requirements first, then compare the model, agent and reasoning setting together.
For a team budgeting automated work, keep benchmark spend separate from engineering time. A low-cost attempt that needs extensive repair and a costly attempt that still fails are different problems. Neither becomes an acceptable release because its partial-progress score looks impressive.
ByteForward has not rerun these tasks. The reported measurements come from Google’s benchmark. The practical takeaway is narrower than a winner declaration. Look for evidence that an agent finishes the type of change you need, not just evidence that it can get close.
Chart and table by ByteForward, adapted from Android Open Source Project benchmark data, under CC BY 2.5. Retrieved October 27, 2026.
