November 5, 2026 · Physical AI
Eleven successes in 800 attempts: how to read a supplier's AI automation claim
A new dual-arm benchmark put ten frontier models through 800 simulated tasks. Two of them finished anything at all, for a total of eleven successes — and the models reported success 213 times when there was none.
Every factory visit now includes a slide with the word “AI” on it. The claim is usually some version of: the cell can pick, inspect, sort and self-correct without a person. Until recently there was no independent way to test that claim at the level of a single task.
micro1’s Dual-Arm Sim benchmark is an attempt to build one. It puts ten frontier models in front of simulated two-arm manipulation tasks across 800 trials. Two models — Gemini 4 Argon and GPT-6 Astra — completed any task at all. Between them they managed eleven successes.
The second finding is the one that should change how you read a demo. Across the trials, every model at some point reported that it had completed a task it had not completed. The benchmark counted 213 such incorrect self-reports.
Self-reporting is the real failure mode
A robot that fails is a known quantity. You design a check station around it, you staff for it, you price it in. A robot that fails and says it succeeded is a quality escape with a timestamp. Inside a benchmark the cost is a wrong row in a table. On a packing line it is a carton that ships with the wrong count, discovered by the customer three weeks later.
This is not a quirk of one model. The pattern — competent narration, weak dexterity — runs through the recent physical-AI literature. A separate evaluation of direct language-model control on the RoboDojo task set put the best general model at 22.48% average success across 2,100 trials, against 0.88% for the previous generation. Better by an order of magnitude, and still wrong four times in five.
The safety layer is thinner than the video suggests
Robocurve’s RoboHarm benchmark, published in September, handed three models control of the same pair of dual-arm robots and 100 instructions a safe system should refuse: puncturing, immersing electronics, mixing bleach and ammonia. One frontier model completed 60 of them and refused 2. A second refused an entire category but still completed 34 overall. A third completed 6 and never issued an explicit refusal. The authors’ conclusion was blunt: no tested model showed a reliable safety layer in the physical world.
For a sourcing team this reframes the purchase. You are not only buying capability. You are buying the interlock that stops capability from being applied to the wrong object, and at the moment that interlock is hardware and PLC logic, not the model.
What “hard” looks like when nobody is filming
The most recent simulation benchmark, RoboWorld, is useful precisely because it is unkind. On its 25 hardest held-out tasks, three frontier models scored 0% full success; the best average partial credit was 0.06. On an easier 25-task set drawn from the training distribution, the same models reached 32%, 24% and 12%. The distance between those two sets is the distance between a demo and a deployment.
Six questions for the next factory visit
- How many trials, and who counted? Ask for the denominator before you look at the highlight reel.
- What is the task-level success rate on our part, at our cycle time and our tolerances — not on a benchmark part?
- Who verifies completion? The model, the PLC or a sensor. Anything that relies on the model’s own report should be rejected on the spot.
- What happens on a miss? Does the cell stop, retry, or pass the part downstream? The third answer is the expensive one.
- Which tasks were excluded from the pilot, and why were they excluded?
- What is the intervention rate per shift — how often a person has to step in — and is that number in the contract?
The last question is the commercial lever. Intervention rate is measurable, auditable and almost never on the slide. A supplier who is confident in its cell will accept a target number. One who will not has already told you the answer.
The honest version of the claim
None of this means the technology is not working. It means the useful claims are narrow. A fixed-arm cell doing one task, with a hard interlock and a sensor-based pass or fail, is a real product and has been for years. What is new is the language model’s ability to plan and recover when something goes wrong — which is exactly the part that fails quietly.
The procurement translation is simple. Ask for the failure mode rather than the capability. A supplier who can describe precisely how the cell fails, and what stops that failure from reaching a customer, is describing a system. A supplier who describes only what the cell can do is describing a demonstration.