Introduction
Planning over long horizons using learned world models is a formidable challenge. As models scale to predict high-dimensional observations across many time steps, optimization becomes ill-conditioned, non-greedy structures create poor local minima, and latent spaces introduce subtle failure modes. The GRASP planner addresses these by lifting trajectories into virtual states, injecting stochasticity, and reshaping gradients. This guide walks you through implementing GRASP for robust, long-horizon planning in your own world model.
What You Need
- A learned world model that predicts future states given current state and action sequences.
- Access to the model's latent representation (e.g., encoder output) and decoder.
- A differentiable optimizer (e.g., Adam) for gradient-based updates.
- An action space (continuous or discrete) and state space (image, latent vector, etc.).
- Hyperparameters: horizon length T, number of optimization iterations, stochasticity scale σ, gradient reshaping factor α.
Step-by-Step Implementation
Step 1: Lift the Trajectory into Virtual States
Instead of optimizing actions directly over the entire horizon, introduce a sequence of intermediate 'virtual states' at each time step. This transformation allows parallel computation across time, breaking the sequential dependency. Formally, replace the single action sequence a1:T with a set of virtual state-action pairs. In practice, create a differentiable buffer of latent states that the world model can jointly predict.
Step 2: Parallelize Optimization Across Time
With virtual states, you can evaluate the objective (e.g., sum of rewards or reconstruction error) for all time steps simultaneously. Use matrix operations to propagate gradients through the entire trajectory in one pass. This avoids the sequential rollout bottleneck and makes long horizons computationally feasible.
Step 3: Inject Stochasticity into State Iterates
Add noise directly to the state iterates during optimization. For each iteration, sample Gaussian perturbations with standard deviation σ and add them to the virtual state estimates. This exploration mechanism helps escape sharp local minima that plague long-horizon planning. Adjust σ as a hyperparameter—too much noise destabilizes, too little fails to explore.
Step 4: Reshape Gradients to Bypass Vision Models
High-dimensional vision models produce brittle gradients that are uninformative for action planning. Replace gradients passing through the vision encoder with a cleaner surrogate. Specifically, compute the gradient of the planning objective with respect to the action, but stop gradients from flowing back through the image encoder. Instead, project the gradient from state space to action space using a learned or fixed Jacobian, effectively reshaping the signal.
Step 5: Iterate the Planning Loop
- Initialize virtual states randomly or from a prior (e.g., current observation).
- Repeat for a fixed number of iterations:
- Compute world model predictions for all time steps using virtual states and candidate actions.
- Evaluate the objective (e.g., negative reward, distance to goal).
- Backpropagate gradients with gradient reshaping (Step 4).
- Update actions and virtual states with an optimizer, adding stochasticity after each update.
- Extract the optimal first action from the converged solution.
Tips for Robust Long-Horizon Planning
- Start with a shorter horizon and gradually increase T during training to avoid catastrophic local minima.
- Anneal the stochasticity scale over iterations—high noise early for exploration, low noise later for fine-tuning.
- Normalize the virtual states to keep them within the world model's training distribution.
- Use multi-step gradient accumulation if GPU memory is limited; virtual states enable recomputation of forward passes.
- Validate with a small number of planning steps before scaling, ensuring gradient reshaping is working correctly.
- Monitor the objective's variance across random restarts—high variance indicates the need for better initialization or more stochasticity.