AA Robotics tests coding agents on physical tasks
Artificial Analysis published its AA-Robotics v0.1 preview on October 14, 2026. Five models control a Hiwonder NexArm using code, four cameras and HTTP.
Eight tabletop tasks test button pressing, toy placement, pouring, boxing blocks, matching shapes, writing hello world, tower building and box measurement.
The report calls attempts zero shot. GPT models use Codex, Claude uses Claude Code and Gemini uses Antigravity, all at high effort.
GPT-6 Astra led with 69.4 to GPT-6.1 Sol’s 64.6. Final states score up to 100, with partial credit. Astra averaged 30 minutes and US$16.36 per task, versus Sol’s US$1.93.
What the score can tell you
A score with partial credit answers how far an attempt progressed under a particular grading rubric. It cannot be read as the percentage of jobs a robot will finish. A useful procurement question is whether the unfinished work can be recovered safely and cheaply, or whether a person must start again.
Consider a hypothetical packing station. Getting several objects into a container might count as measurable progress in an experiment. If the final container cannot be closed, the customer still cannot receive the order. A buyer should decide which intermediate steps have value and which failures invalidate the entire job before choosing a score to optimize.
This is the same reading discipline that matters in coding benchmark comparisons. Check what earns credit, what invalidates a result and how the headline number was assembled.
The controller is part of the comparison
Different harnesses mean this compares complete agent setups, not isolated model quality.
The practical question for a robotics team is which combination it can actually operate. A different tool interface, camera arrangement or movement library may change what an agent can attempt. To attribute a gain to one component, an evaluator should hold the other components steady or explicitly test their contribution. Record software revisions and settings so a later result can be compared fairly.
Hiwonder’s own tutorial describes a separate development route using human demonstrations. An operator moves a leader arm while a follower arm performs the task, and the system records sensor data and commands. Those demonstrations support training a control policy. The tutorial also identifies five joint servos plus the gripper.
That documentation is useful platform context. It does not establish that the benchmark used the demonstration training workflow. A controller trained for a particular operation and a general agent encountering a new instruction answer different engineering questions. Each deserves evaluation against its intended job.
Reliability needs repeated trials
The report does not specify repeat trial counts or confidence intervals.
NIST’s assembly benchmark guidance offers a useful reference for stronger evidence. Its task board procedure calls for repeated trials with new randomized placements and statistical analysis. It separates task completion time from reliability and encourages measurements for individual operations as well as the whole board. It also cautions that one task board cannot fully characterize a robot.
For this preview, the next questions are how often each task succeeds across fresh attempts, how much results vary and whether the ranking survives changed starting conditions. Ask for the number of independent runs, the reset procedure, the distribution of scores and an appropriate uncertainty estimate. Multiple movements inside one attempt should not be mistaken for independent repetitions of the complete task.
That is a request for further evidence, not a claim that unreported testing never happened. A small difference between averages deserves restraint until the underlying variability is known.
Budget for a finished workflow
Before connecting an agent to a production process, define a completed unit of work and measure the full route to it. Include setup, supervision, retries, recovery, maintenance and the equipment needed to operate safely. Keep those prospective costs separate from model usage charges. ByteForward has not measured a deployment budget for these systems.
A sensible pilot would begin with one bounded operation, explicit stopping conditions and a clear route for human intervention. Track completed work, elapsed time, damaged or lost material and the effort needed to recover from failure. Expand the test to unfamiliar objects or changed conditions only when the earlier stage is understood.
The useful next step is a reproducible evaluation matched to an actual job. A tabletop result alone cannot establish general competence in homes or factories, or show that a robot is safe to deploy around people. ByteForward has not run this benchmark.
Illustrative photograph of robotic arms and grippers, published in 2019. The image does not depict the NexArm used in the AA Robotics benchmark. Photo by Franck V. on Unsplash under the Unsplash License.
