SSG: Scaled Spatial Guidance for Multi-Scale Visual Autoregressive Generation

Visual autoregressive (VAR) models generate images through next-scale prediction, naturally achieving coarse-to-fine, fast, high-fidelity synthesis mirroring human perception. In practice, this hierarchy can drift at inference time, as limited capacity and accumulated error cause the model to deviate from its coarse-to-fine nature. We revisit this limitation from an information-theoretic perspective and deduce that ensuring each scale contributes high-frequency content not explained by earlier scales mitigates the train-inference discrepancy. With this insight, we propose Scaled Spatial Guidance (SSG), training-free, inference-time guidance that steers generation toward the intended hierarchy while maintaining global coherence. SSG emphasizes target high-frequency signals, defined as the semantic residual, isolated from a coarser prior. To obtain this prior, we leverage a principled frequency-domain procedure, Discrete Spatial Enhancement (DSE), which is devised to sharpen and better isolate the semantic residual through frequency-aware construction. SSG applies broadly across VAR models leveraging discrete visual tokens, regardless of tokenization design or conditioning modality. Experiments demonstrate SSG yields consistent gains in fidelity and diversity while preserving low latency, revealing untapped efficiency in coarse-to-fine image generation. Code is available at https://github.com/Youngwoo-git/SSG.

💡 Research Summary

The paper addresses a persistent problem in visual autoregressive (VAR) models: although these models are designed to generate images in a coarse‑to‑fine, multi‑scale fashion, during inference they often drift from this hierarchy. Limited model capacity and error accumulation cause later scales to redundantly reproduce low‑frequency information already captured, rather than adding the intended high‑frequency details. This “train‑inference discrepancy” degrades image fidelity and spatial coherence.

To understand and remedy this, the authors reinterpret VAR sampling through the lens of the Information Bottleneck (IB) principle, but with a reversed objective. Instead of compressing an input, each generation step should maximize the mutual information between the newly generated residual and the final image while minimizing redundancy with the previously generated state. By decomposing the final image into low‑frequency (L) and high‑frequency (H) components, the IB objective simplifies to maximizing I(z_k; H( f̂_K )) – I(z_k; L( f̂_K )). In practice, the model predicts logits ℓ_k for the current token map; the authors define a “semantic residual” Δ_k = ℓ_k – ℓ_prior, where ℓ_prior is a coarse‑scale prior derived from the previous step.

The core contribution, Scaled Spatial Guidance (SSG), adds a scaled version of this residual to the raw logits: ℓ_SSG_k = ℓ_k + β_k · Δ_k. The guidance scale β_k controls the trade‑off between injecting new high‑frequency detail and preserving the base model’s coherence. The authors show that this quadratic objective has a closed‑form maximizer, making SSG computationally trivial.

A crucial component is how to construct ℓ_prior without introducing artifacts. Simple spatial interpolation either over‑smooths (linear) or creates blocky high‑frequency noise (nearest‑neighbor). The authors propose Discrete Spatial Enhancement (DSE), a frequency‑domain upsampling technique. DSE applies a discrete cosine transform (DCT) to both the original coarse logits and a linearly upsampled version. Low‑frequency coefficients from the original are retained, while high‑frequency coefficients from the upsampled version are inserted, yielding a hybrid spectrum that is then inverse‑transformed back to the spatial domain. This process preserves the exact low‑frequency structure while providing a plausible high‑frequency extrapolation, ensuring that Δ_k truly isolates novel detail.

Implementation-wise, SSG operates entirely on the logits at inference time, requiring no changes to model weights, no additional forward passes, and only a few extra DCT/IDCT operations. Consequently, the latency overhead is negligible, preserving the speed advantage of VAR models (typically a dozen steps).

Experiments evaluate SSG on several state‑of‑the‑art VAR systems with diverse tokenizers (hybrid, bit‑wise) and conditioning modalities (class‑conditional, text‑conditional). Across all settings, SSG consistently improves quantitative metrics: lower FID, higher IS, and reduced LPIPS, indicating sharper high‑frequency content. Qualitative examples show clearer fine structures such as a bird’s beak or intricate textures that the baseline fails to render. Diversity metrics remain stable or slightly improve, demonstrating that SSG does not collapse the output distribution. Compared to recent diffusion models and masked‑prediction refiners, SSG achieves comparable or better quality while retaining the low‑step count and minimal compute.

In summary, the paper provides a theoretically grounded, training‑free guidance mechanism that enforces the intended coarse‑to‑fine hierarchy in VAR generation. By explicitly encouraging each scale to contribute novel high‑frequency information via the semantic residual and by constructing a reliable prior through DSE, SSG bridges the train‑inference gap without incurring significant computational cost. The method is model‑agnostic, easy to integrate, and opens a practical path for improving existing VAR pipelines in both research and production environments.