Researcher runs 600 tests exposing flaws in JEPA world-model benchmarks
An independent researcher built a memory-graph planner (LeWM) and tested it against the standard Push-T benchmark, finding that 84% of the benchmark's test goals already closely matched states recorded elsewhere in the training demonstrations, making the task easier than it appears. The researcher also found results were highly sensitive to sample size and random seed, with success rates on the official 50 episodes dropping from 68% to 44% on a larger 250-episode set, and 38 episodes flipping outcome purely from a changed seed.
GoKawiil's interpretation of the reporting above, not reported fact.
The findings suggest that a widely used robotics benchmark may overstate how well planning models generalize, since many of its 'test' scenarios are not meaningfully different from training data. This could prompt researchers to treat reported success rates on Push-T and similar benchmarks with more skepticism, and to favor tougher cross-demonstration tests like the one proposed, which produced a notably lower 45% success rate.
- 84% of standard Push-T test goals nearly duplicated states already present in training demonstrations.
- Success rates varied sharply with sample size (68% vs 44%) and random seed (38 of 250 episodes flipped).
- A harder benchmark requiring travel between unconnected demonstration states dropped success from 75% to 45%.
Source: haddyalchaer.com — Haddy Alchaer, 2026-11-13
Published there as: “After 600 experiments, here's what I discovered about JEPA world models”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.