Beyond Task Success: Measuring Repair in Vision-Language-Action Policies

1. Introduction

Ordinary benchmark success asks whether a policy can finish a task. Repair asks a more constrained question: after part of the task is already correct, can the policy fix the remaining failure while preserving the achieved conditions? The distinction matters when a controller resumes in a state that differs from the training distribution's usual episode starts. MissionOS therefore measures target improvement and preservation separately, and does not equate policy inference or dispatch with successful repair. [1]

The public evidence spans several protocols. The primary cross-executor comparison uses a digest-bound diagnostic clone of the same displaced moka-pot scene for Cosmos Policy and VLA-0. GR00T's native cohort and an additional book-to-caddy task answer related but different questions. Keeping these experiments separate prevents a misleading combined leaderboard.

2. Methods: Admit a Recoverable Failure Before Testing Repair

The aligned fixture uses LIBERO-10's task to place both moka pots on the stove. A 3 cm displacement leaves the actual predicate vector at [true, false, true]: one target condition fails while the other pot and stove condition remain satisfied. A 2 cm displacement was rejected because the task remained successful; a tested 5 cm fixture was unstable. The selected 3 cm state remained a failure after restoration and settling. It also changed 3,565 pixels in the processed primary camera and 6,595 in the wrist camera, reducing concern that the perturbation was visually negligible. [1]

A privileged scripted controller establishes recoverability through the same original 7D simulator action interface. It can inspect simulator object state, so its success is a positive control, not learned-model performance. On the exact fixture, it contacted the target after action 32, reached all predicates after action 33, and retained them for 20 stationary steps in three of three fixed-condition replays.

Both learned policies used their pinned checkpoints, the original instruction, no additional training, and a 128-action ceiling. Actual LIBERO predicates determined success. Generated future images could not establish an observed effect. Evaluation distinguished target engagement, entry into the complete predicate conjunction, and stable completion over a subsequent 20-step stationary hold. The last criterion tests whether a momentary success remains valid after corrective motion ends.

3. Results: Contact Is Not Stable Completion

Cosmos moved the target by approximately one nanometer and retained [true, false, true]. Its action outputs were not uniformly zero: earlier pose and language probes changed action values. Action sensitivity therefore did not translate into corrective target engagement in this bounded trial. [1]

VLA-0 first contacted the target after actions 66 and 70 in its first two trials, entering the full conjunction after actions 84 and 83. Those trials had stopped at first entry. Their recorded actions were consequently replayed from the same bound snapshot and followed by the stability hold. Both reproduced the conjunction, retained it for four full hold steps, and lost the repaired predicate on step five. The protected pot stayed within its fixed 5 mm bound. These were action replays, not additional policy trials.

A third, fresh policy trial contacted after action 67 and moved the target by as much as 14.06 cm, but never reached the conjunction within 128 actions. This observation adds variability to the positive targeting result. It does not rescue the stability claim.

Separate GR00T and second-task evidence

GR00T N1.7's native same-world cohort observed target improvement in 0/5 natural-failure loops, each with one approved attempt and no automatic retry. Two loops targeted predicate 0 and three targeted predicate 1. The single-attempt protocol was amended after the first observation and fixed before loops 2–5; the originally planned maximum-two-attempt protocol remains unmeasured. Each loop preserved its own contract-bound conditions. [2]

In the book-to-caddy task, a displaced and settled book lay 5.314 cm from its successful source position. A privileged controller achieved stable success in 3/3 runs, using 33 actions including the hold. Cosmos passed the ordinary task after 166 actions, but made no contact during the displaced-state trial's 128 actions. VLA-0 failed the ordinary-task admission gate at 520 actions, so its book-repair stage was not run. That is an unmeasured repair condition, not a repair failure. [3]

4. Discussion

The aligned comparison supports an executor-dependent difference in corrective targeting: VLA-0 repeatedly interacted with the failed object, whereas Cosmos did not in its tested run. It also shows why a first-success stopping rule can overstate achievement. A conjunction reached during motion can disappear when the system settles; stable completion needs its own observation window.

The privileged control excludes infeasibility under the tested interface and budget for the admitted fixture. It does not provide an equal-information competitor, since it receives object-state access unavailable to the camera-conditioned policies. The controlled comparison helps identify where behavior breaks down without proving a training-data or architecture-level cause.

5. Limitations

The aligned trials use one task, initialization, seed, fixture, and checkpoint configuration. They are diagnostic MuJoCo clones, not live same-world MissionOS repair loops. No independent controller acknowledgement or physical Panda execution was established. The GR00T native cohort uses different failures and cannot be ranked directly against those clone trials.

An older VLA-0 experiment displaced a pot by 22.7 cm and observed no sustained target-directed approach through 520 actions, despite oracle recoverability. It remains valid historical evidence for that configuration, but does not override the later 3 cm findings. Neither study estimates a general repair rate. Extending the evidence requires independently admitted recoverable failures, strong nominal controls, and a stability criterion fixed before policy evaluation.

6. Conclusion

The strongest result is a measured separation of capabilities: recoverability, target contact, predicate entry, and stable repair are distinct. The tested VLA-0 configuration demonstrated corrective targeting on the aligned fixture; stable completion was not established. Preserving that boundary makes both promising behavior and bounded negative results useful for the next experiment.