FRAPPE: Infusing World Modeling into Generalist Policies via Multiple Future Representation Alignment

Enabling VLA models to predict environmental dynamics, known as world modeling, has been recognized as essential for improving robotic reasoning and generalization. However, current approaches face two main issues: 1. The training objective forces models to over-emphasize pixel-level reconstruction, which constrains semantic learning and generalization 2. Reliance on predicted future observations during inference often leads to error accumulation. To address these challenges, we introduce Future Representation Alignment via Parallel Progressive Expansion (FRAPPE). Our method adopts a two-stage fine-tuning strategy: In the mid-training phase, the model learns to predict the latent representations of future observations; In the post-training phase, we expand the computational workload in parallel and align the representation simultaneously with multiple different visual foundation models. By significantly improving fine-tuning efficiency and reducing dependence on action-annotated data, FRAPPE provides a scalable and data-efficient pathway to enhance world-awareness in generalist robotic policies. Experiments on the RoboTwin benchmark and real-world tasks demonstrate that FRAPPE outperforms state-of-the-art approaches and shows strong generalization in long-horizon and unseen scenarios.

💡 Research Summary

FRAPPE (Future Representation Alignment via Parallel Progressive Expansion) addresses two fundamental shortcomings of existing visual‑language‑action (VLA) policies that attempt world modeling: an over‑reliance on pixel‑level reconstruction, which hampers semantic learning and out‑of‑distribution (OOD) generalization, and the accumulation of errors when predicted future observations are directly used during inference. The authors propose a two‑stage fine‑tuning paradigm that injects world‑modeling capability into a diffusion‑based robotic policy (RDT) while keeping computational overhead manageable.

In the mid‑training stage, a learnable “future prefix” token is concatenated to the standard RDT input (proprioception, noisy actions, language instruction). The model is trained to predict both the future action chunk and the future prefix. Instead of supervising the prefix with raw pixel reconstructions, the authors employ multiple visual foundation models (VFMs) – CLIP, DINOv2, and ViT – as teachers. The teacher encoders generate latent embeddings of future observations; a cosine‑similarity loss aligns the model’s predicted prefix with these embeddings, using a stop‑gradient to keep the teachers frozen. This forces the policy to learn a compact, semantic representation of future states, reducing the need for pixel‑level detail.

The post‑training stage introduces Mixture‑of‑Prefix‑and‑LoRA (MiPA). The shared RDT backbone is duplicated into M expert streams, each equipped with its own prefix and Low‑Rank Adaptation (LoRA) modules. Each expert aligns with a distinct VFM teacher, and a lightweight router learns gating weights wᵢ to combine the experts’ latent action representations zᵢ. The final action is produced by a shared MLP head applied to the weighted sum Σ wᵢ·zᵢ. To prevent a single expert from dominating, a load‑balancing loss encourages uniform gating logits, and a smoothing term guarantees a minimum weight ε for every expert. Only the prefixes, LoRA adapters, and router parameters are trainable, keeping the memory footprint low while allowing parallel scaling.

A key practical advantage is the ability to leverage large‑scale, action‑free egocentric human videos. Because the future‑prefix prediction does not require action labels, the method can ingest massive human manipulation footage, dramatically reducing the reliance on costly tele‑operation data. In low‑data regimes (e.g., 120 trajectories per hour of expert tele‑op), FRAPPE improves performance by 10–15 % over a tele‑op‑only baseline.

Experiments on the RoboTwin 2.0 benchmark (both simulated long‑horizon tasks and real‑world manipulation) demonstrate that FRAPPE consistently outperforms state‑of‑the‑art diffusion policies (RDT, FLARE, UD‑VLA) and specialized world‑modeling approaches (e.g., FLARE). The gains are most pronounced in long‑horizon, OOD, and data‑scarce settings, confirming the effectiveness of multi‑teacher alignment and the two‑stage training schedule.

Strengths of the work include: (1) a principled shift from pixel reconstruction to semantic latent alignment, (2) a scalable expert‑router architecture that mitigates mode collapse via explicit load‑balancing, (3) data efficiency through action‑free video pre‑training, and (4) modest parameter overhead thanks to prefix‑and‑LoRA fine‑tuning. Limitations involve sensitivity to the choice and quality of VFM teachers, and increased inference compute when many experts are active, which may require further optimization for real‑time control.

Overall, FRAPPE presents a compelling new paradigm for integrating world modeling into generalist robotic policies, achieving superior scalability, data efficiency, and robust generalization, and it opens promising avenues for future research on adaptive teacher selection and lightweight routing mechanisms in embodied AI.