H-JEPA · Animation study

From goals to subgoals to actions

A two-level example of hierarchical planning. Follow the sequence, or select a step to inspect it.

Two-level hierarchical planning Encode current and goal observations at both levels. Optimize level-2 macro-actions to approach the goal. Use the first level-2 prediction as a subgoal. Optimize two level-1 primitive actions by comparing their final prediction, lifted through the level-2 encoder, with that subgoal. Execute an action prefix, observe, and replan. Two levels · stride s₂ = 2 · state window w₂ = 1 · subgoals K₂ = 1 LEVEL 2 LEVEL 1 o₀Current observation ogGoal observation E¹E² E¹E² z₀⁽¹⁾z₀⁽²⁾ g⁽¹⁾g⁽²⁾ F²F² ⋯ ẑH₂(2) C² a₀⁽²⁾ aH₂−1(2) Predicted coarse trajectory Match the final prediction to the goal C² = ‖ẑH₂(2) − g⁽²⁾‖² Optimize a₀⁽²⁾, …, a⁽²⁾H₂−1 ẑ₁⁽²⁾ Subgoal C¹ Predicted fine trajectory F¹F¹ ẑ₁⁽¹⁾ẑ₂⁽¹⁾ a₀⁽¹⁾a₁⁽¹⁾ E² Lift to level 2 Match the lifted prediction to the subgoal C¹ = ‖E²(ẑ₂⁽¹⁾) − ẑ₁⁽²⁾‖² Optimize a₀⁽¹⁾, a₁⁽¹⁾ Execute an action prefix Use the optimized primitive actions ObserveReplan from the new state

The current observation and goal are encoded progressively through the hierarchy, producing an initial state and a goal representation at every level.

0:00 / 0:18
Latent statesActions being optimizedPlanning objectives

Adapted from the original TikZ schematic. E denotes an encoder; F denotes a predictor. This preview illustrates the planning sequence, not measured optimization trajectories.