This is great!
Really nice (brief) description on how gains were made.
To level set this:
1 failure in 50 gets a human fired by EOW.
Good work! (not a product yet)
Two months of ablating π0.5 finetunes on a real manufacturing task. The policy now succeeds 98% of the time.
We're publishing all results, all the data, and every run, including the ones that went nowhere.
What we learned:
1/ 5x more data took us from 63% to 76%. Pure scaling was the weakest lever we found.
2/ Diversity got more out of the same hours. 4h spread across five scenes beat 4h in the eval scene by 30pp.
3/ Quality did even better. One clean, slower hour on top of the 21h: 76% → 90%. That single hour added more than the 17 hours before it.
4/ So we tried quality on its own: 1h of clean data plus 240 rollouts with human interventions. 1.7h in total, 28% → 88%. That's 12 points above the 21h model with a twelfth of the data.
5/ Inference settings add more on top, with no retraining. They took the 21h model from 76% to 93%, and the 21h+1h model from 90% to 98%.
Just as useful: what didn't matter. Learning rate, batch size, relative vs. absolute joints, image augmentation. All less important than we expected.
Full write-up linked below.