Researcher runs 600 tests exposing flaws in JEPA world-model benchmarks
An independent researcher built a memory-graph planner (LeWM) and tested it against the standard Push-T benchmark, finding that 84% of the benchmark's test goals already closely matched states recorded elsewhere in the training demonstrations, making the task easier than it appears. The researcher also found results were highly sensitive to sample size and random seed, with success rates on the official 50 episodes dropping from 68% to 44% on a larger 250-episode set, and 38 episodes flipping outcome purely from a changed seed.