A series of 600 experiments on the LeWM (Latent World Model) revealed that a JEPA world model's success on long-horizon tasks is limited more by its planning architecture than by its ability to predict physics. While the model struggled to solve 75-step tasks with a 22% success rate in its vanilla form, researchers found that breaking these tasks into smaller 25-step segments boosted success to 80% without any retraining of the underlying model.
The JEPA Architecture and the Prediction Gap
The Joint-Embedding Predictive Architecture (JEPA) operates by converting visual observations into compact vectors called embeddings. Unlike generative models that attempt to predict every pixel in a future frame, JEPA uses an encoder to extract relevant features, a predictor to forecast the next embedding based on an action, and a planner to evaluate potential action sequences.
This figure explains the fundamental architecture of the JEPA model, which uses an encoder, predictor, and planner.
In testing the LeWM model on the Push-T benchmark—where a robot must move a T-shaped block to a target—researchers found that the model's predictions are nearly as accurate as perfect physics over short horizons of 25 steps. This suggests that the primary bottleneck for long-horizon tasks is not the inaccuracy of the world model itself, but how the planner handles compounding errors over 75 steps.
This image illustrates the Push-T benchmark used to evaluate the model's planning and prediction capabilities.
Chart drawn by HN Digest from figures reported by haddyalchaer.com; values are shown as written in the source.
“LeWM's useful planning horizon is about 25 steps, so it was clear that the way to go further is to chain segments of that length together.”
— After 600 experiments, here's what I discovered about JEPA world models
Hierarchical Planning and Memory Graphs
One significant improvement involves hierarchical planning. By dividing a 75-step goal into three 25-step segments, the model stays within its 'accuracy zone,' where it can successfully predict the next state. This method outperformed the vanilla model by 58 percentage points, nearly matching the performance of a planner provided with perfect expert subgoals.
Another method utilizes a latent memory graph built from 460,000 embedded states of past expert demonstrations. By treating these states as a map and adding a cost to 'jumps' between different demonstrations, the planner can navigate toward a goal by chaining together segments of successful past behavior. This approach achieved an 85% success rate on long-horizon tasks.
This visual demonstrates the method of using a latent memory graph to navigate toward goals by chaining past behaviors.
Refining Goal Scoring and Limitations
Research also highlighted a flaw in how goals are scored. Because a goal image includes the robot's arm, the planner often fails by trying to perfectly replicate the arm's position rather than focusing on the block's location. By adjusting the scoring to ignore the arm, success rates on certain tasks increased by 30 points.
Despite these gains, a significant challenge remains: generalizing to tasks that lack perfect expert demonstrations. While memory graphs and transformations (like rotating or mirroring past data) help fill gaps, the model still struggles when a goal requires navigating through states that have never been captured in a demonstration.