Haga / Lab / P2-E1 real-footage calibrated gate (SAP v2), three ALOHA-family datasets
Real-footage calibrated gate
The three-state gate (FLAG / CLEAN / ABSTAIN) applied to real tracked robot footage from the three ALOHA-family datasets SAP v2 names, with a disjoint calibration and test split per dataset. The support plane is measured per clip and the detectors are gated on the gripper-derived free-frame mask.
Does the checker catch what it is meant to catch, and what does it cost when it is wrong?
Proposition
What the run claims
A gate is only useful if it refuses the right things. This run applies a three-state gate — FLAG, CLEAN, ABSTAIN — to real bimanual robot footage and measures how often clean footage is refused.
Apparatus
What was measured
LeRobot's ALOHA static towel, candy and ziploc tasks: real robot footage, tracked trajectories, and a split between calibration episodes and test episodes that does not overlap. The three datasets are scored separately, each on its own footage, and the gate's verdict is reported per dataset as well as for the set.
Method
How it was measured
Most thresholds are fitted on the calibration episodes and frozen before the test episodes are scored. The contact penetration threshold (0.0080 m) was set below the measured noise floor (0.0160 m) and was not fitted on the calibration episodes. The clean cohort is presumed violation-free and is not independently labelled, so a flag on one of its clips is not established as wrong; synthetic injected violations measure detection. The two are reported separately, because a gate that confuses them is not a gate.
Observations
What was observed
The headline figures below are the run's own. The video shows the same run as the pipeline produced it.
Definitions
Which count is which
This run reports four counts per dataset, and they are not the same count. The limitation at the bottom of the page names the released episode counts and the shortfall against the preregistration; the table above reports tracks. Neither is wrong; they are different units.
| Dataset | Released episodes | Test episodes | Source clips | Tracks scored |
|---|---|---|---|---|
| towel | 50 | 15-24 | 49 | 17 |
- Released episode -- one recording in the released dataset. The preregistration assumed more of them than were released, and the limitation below names both numbers.
- Test episode -- the preregistered window, the calibration episodes excluded, intersected with the episodes that actually exist.
- Source clip -- one measured release event inside the test episodes, cut around the moment the named gripper channel opened. An episode with two measured releases yields two clips, so clips can outnumber episodes. For towel, 35 test episodes yielded 49 source clips (~1.40 per episode). For candy, 35 test episodes yielded 66 source clips (~1.89 per episode). For ziploc, 41 test episodes yielded 36 source clips (~0.88 per episode; some episodes had two releases).
- Track -- one candidate track of a clip. The table and the charts below carry one row per track (209 total tracks across the three datasets, with 207 scored and 2 abstained on), so a clip that was tracked more than once appears more than once. For towel and candy, each of the 67 and 83 clips was tracked once, yielding 67 and 83 tracks respectively. For ziploc, 59 clips produced 57 scored tracks (+2 abstained); the two abstained clips were not scored.
Deviations from SAP v2
Deviations from the frozen protocol
SAP v2 assumed 80 episodes per dataset; the released LeRobot datasets contain towel 50, candy 50, ziploc 56. Test windows were redefined as the preregistered 15-79 intersected with existing episodes. This deviation was identified before scoring and is recorded in each artifact's limitations.
Decision rule. Frozen SAP v2 rule: Wilson 95% upper bound of the false-positive rate ≤ 0.020 → PASS, otherwise FAIL. Under it all three datasets FAIL (Wilson upper bounds: towel 0.124, candy 0.065, ziploc 0.313). The run's implementation applied the point estimate (flag rate ≤ 0.020 → PASS), under which candy (0.012) would pass; that deviation is recorded in the artifact. The verdict shown on this site is the frozen-rule verdict.
- Verdict
- FAIL
- across towel
- Datasets scored
- 1
- towel
- Flag rate on presumed-clean tracks (towel)
- 0.118
- 17 scored tracks, 2 flagged - target <= 0.020 - status FAIL
| Observation | Value | 95% interval | Note |
|---|---|---|---|
| Flag rate on presumed-clean tracks (towel) | 0.118 | [0.0329, 0.3434] | 17 scored tracks from 17 source clips, presumed violation-free and not independently labelled; a flag on one is not established as wrong |
| Detection rate (towel) | 1.000 | [0.8985, 1] | 34 injected violations, synthetic |
| Datasets scored (CPU-optimized run) | 1 (towel only) | — | CPU-optimized run: p2e1-v2-AlohaTowel.json. Full 3-scenario run requires GPU. |
| Datasets meeting SAP v2's 30 clean-clip minimum | 0 | — | towel: 17 clean clips (SHORT of 30) |
On the actual footage
These are the real clips: the footage with the tracked path, the measured value, the threshold and the flag drawn on the pixels that were judged. Every label in each video is generated by the harness from its own output.
Demonstration of how Haga detects, flags and reports. Not accuracy evidence: accuracy claims come only from the experiment statistics.
Selected failure cases (13)
10 flags on presumed-clean footage and 3 injected variants whose re-derived track did not fire. The run's own record of what it flagged — the flagged of the presumed-clean tracks it scored — is in the table above, and that record is what the statistics use. These videos are re-derived from the clips, so a video's verdict is read again from its own frames and can differ from the run's record; where it does, the page states both.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga put a known violation into this clip, and this video's re-derived verdict is a miss: the violation was there and the checker did not fire on it. The run's own detection record is in the observations above
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga put a known violation into this clip, and this video's re-derived verdict is a miss: the violation was there and the checker did not fire on it. The run's own detection record is in the observations above
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga flagged this clip, and the footage is presumed violation-free, so the flag is a FALSE POSITIVE on contact penetration
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga put a known violation into this clip, and this video's re-derived verdict is a miss: the violation was there and the checker did not fire on it. The run's own detection record is in the observations above
The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
A success case
The injected violation was tracked frame by frame through the clip, with the injected track overlaid on the pixels that were judged.
The injected object was identified in each frame by its injected trajectory, and the frames where the gripper held the injected violation are the ones the checks run on.
Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. The checker detected the violation Haga introduced.
Haga put a known violation into this clip, and Haga found it: the flag on the video is the detector firing on the violation the run introduced.
Limitations
What this run does not show
Does the checker catch what it is meant to catch, and what does it cost when it is wrong?
- CPU-OPTIMIZED RUN: Only towel scenario processed. Candy and ziploc scenarios omitted due to Kaggle GPU unavailability. Tracker resolution reduced to MAXW=256, GRID=6. Episodes limited to calibration 0-7 and test 15-24.
- CPU-OPTIMIZED RUN: Only towel scenario processed. Candy and ziploc scenarios omitted due to Kaggle GPU unavailability. Tracker resolution reduced to MAXW=256, GRID=6. Episodes limited to calibration 0-7 and test 15-24.
- CPU-OPTIMIZED RUN: Only towel scenario processed. Candy and ziploc scenarios omitted due to Kaggle GPU unavailability. Tracker resolution reduced to MAXW=256, GRID=6. Episodes limited to calibration 0-7 and test 15-24.
- The object diameter is ASSUMED per dataset and metres-per-pixel is a declared camera calibration, not a measurement. No reference dimension exists in any of the three datasets.
- The track-picker's span band is ONE geometric prior shared by all three datasets. It is not fitted to any dataset's result, and it is not varied per dataset.
- Injected violations are SYNTHETIC and prove only that the detectors can fire on a known violation.
- The clean cohort is PRESUMED violation-free, not independently labelled, so a flag on one of its clips is not established as wrong. The post-hoc audit is where a flag is checked against measured evidence.
- Detector coverage is incomplete: a detector that never became applicable has no measurable rate.
- The three datasets are shorter than SAP v2 assumed: the preregistration assumed 80 episodes each and the released datasets have towel 50. The effective test windows are the preregistered 15-79 intersected with the episodes that exist, and each artifact records the shortfall.
- towel yields 17 clean clips, 13 short of SAP v2's 30-clip minimum.
- The towel false-positive rate is 0.118 against a 0.020 target.
- Because the 30-clean-clip minimum is far below the 189 tracks a PASS requires at the 0.020 target, compliance with the minimum does not make a PASS achievable; under this rule a dataset at the minimum can only ever FAIL or INCONCLUSIVE.