Lab
Haga is one gate in a pipeline you already run.
The pipeline below is the order of operations in a release: artifact in, decision out. Haga occupies the middle — after tracking, before the decision is written. It does not replace your evaluation suite, your reviewers, or your sign-off. It adds an adversarial pass that is run the same way every time.
- trackTrajectories are extracted from the artifact. In simulation, from the episode logs; in video, from the tracked points.1
- identifyThe task, the stress tier and the applicable checks are determined, so the same criteria are applied to every run.2
- determineSuccess or violation is measured against a criterion frozen before the run, never after seeing the result.3
- flagThe measured value is compared to the gate threshold, and the verdict is one of three states: flag, clean, abstain.4
- reportThe artifact is written and stored, so the run can be audited later.5
Experiments
Published runs
Every run below states its method, its measurement, its interval, and its limits.
Policy stress in simulation
CONTROLP1-E1 policy stress (robosuite, SAP v2)
A scripted OSC_POSE position controller in robosuite, stressed by tiered mass and friction and scored against a pre-registered relative gate. Simulation only: no learned policy, no real robot.
Control run:a scripted controller, so this is a harness check rather than a policy result. Lift holds at 1.000 under every stress tier, which is the expected outcome for a controller that cannot degrade.
Real-footage calibrated gate
FAILP2-E1 real-footage calibrated gate (SAP v2), three ALOHA-family datasets
The three-state gate (FLAG / CLEAN / ABSTAIN) applied to real tracked robot footage from the three ALOHA-family datasets SAP v2 names, with a disjoint calibration and test split per dataset. The support plane is measured per clip and the detectors are gated on the gripper-derived free-frame mask.
VerdictFAILacross towel
Under the frozen rule a PASS at a 0.020 target needs at least 189 clean tracks with zero flags (280 with one); each dataset here has fewer (67, 83, 57), so none could pass.
Because the 30-clean-clip minimum is far below the 189 tracks a PASS requires at the 0.020 target, compliance with the minimum does not make a PASS achievable; under this rule a dataset at the minimum can only ever FAIL or INCONCLUSIVE.
Generative world-model cohort
FAILP2-E2 generative cohort (frozen thresholds)
The frozen-threshold physics-consistency checker scored over three world models' held-out generations, against real control clips. Sensitivity and specificity are reported together; a specificity number is published only when the real-control cohort is large enough to have a usable denominator.
cogvideox flagged0.8898 of 9 held-out clips flagged
Superseded checker (binary FLAG/CLEAN, no ABSTAIN). Re-scoring this cohort under SAP v2 is outstanding; until then its flag rates carry no information (real-control specificity 0.000, 15/15 flagged).
Methods
What SAP v2 freezes
SAP v2 is the Scenario Assessment Protocol — Haga's verification protocol, version 2. The frozen document carries only the letters, so this page spells them out: the scenarios a run is scored on and the assessment — endpoints, thresholds and gates — it is scored by.
Version 2 replaced an unverifiable version 1: a v1 result may appear only alongside the artifact that reproduces it. The v1 policy-stress ladder has no such artifact, so it is not published anywhere on this site. v2 runs are fresh, pre-registered and independently verifiable. It freezes three things.
- Policy stress (P1). Four tasks, 50 held-out seeds per task, four tiers, the same seeds paired across tiers, and a relative gate: a tier passes when it keeps at least 80% of that task's own nominal success rate. The controller is scripted on purpose, so a difference between tiers is a difference in condition and not a difference in learning.
- Real-footage gate (P2-E1). A three-state gate — FLAG, CLEAN, ABSTAIN — over real tracked robot footage. The support plane is measured per clip rather than declared, the detectors are gated on the gripper-derived free-frame mask. The contact penetration threshold (0.0080 m) was set below the measured noise floor (0.0160 m) and was not fitted on the calibration episodes. Other thresholds were fitted on calibration episodes disjoint from the test episodes.
- Generative cohort (P2-E2). Scored under frozen thresholds over held-out generations from three models, against 15 real control clips. Published FAIL (real-control specificity 0.000, 15/15 flagged) — the generative cohort is measured and published.
The three datasets are shorter than SAP v2 assumed: the preregistration assumed 80 episodes each and the released datasets have towel 50, candy 50, ziploc 56. The effective test windows are the preregistered 15-79 intersected with the episodes that exist, and each artifact records the shortfall.
Every number in the lab comes from a Kaggle run under this protocol, and every run ships its limitations with it. Where a run failed, the failure is published with the same prominence as the result.
Deviations from SAP v2
Deviations from the frozen protocol
SAP v2 assumed 80 episodes per dataset; the released LeRobot datasets contain towel 50, candy 50, ziploc 56. Test windows were redefined as the preregistered 15-79 intersected with existing episodes. This deviation was identified before scoring and is recorded in each artifact's limitations.