Dynamic Frequency Modulation for Controllable Text-driven Image Generation

The success of text-guided diffusion models has established a new image generation paradigm driven by the iterative refinement of text prompts. However, modifying the original text prompt to achieve the expected semantic adjustments often results in unintended global structure changes that disrupt user intent. Existing methods rely on empirical feature map selection for intervention, whose performance heavily depends on appropriate selection, leading to suboptimal stability. This paper tries to solve the aforementioned problem from a frequency perspective and analyzes the impact of the frequency spectrum of noisy latent variables on the hierarchical emergence of the structure framework and fine-grained textures during the generation process. We find that lower-frequency components are primarily responsible for establishing the structure framework in the early generation stage. Their influence diminishes over time, giving way to higher-frequency components that synthesize fine-grained textures. In light of this, we propose a training-free frequency modulation method utilizing a frequency-dependent weighting function with dynamic decay. This method maintains the structure framework consistency while permitting targeted semantic modifications. By directly manipulating the noisy latent variable, the proposed method avoids the empirical selection of internal feature maps. Extensive experiments demonstrate that the proposed method significantly outperforms current state-of-the-art methods, achieving an effective balance between preserving structure and enabling semantic updates.

💡 Research Summary

The paper addresses a fundamental limitation of text‑conditioned diffusion models: when a user slightly modifies a prompt to achieve a specific semantic change (e.g., altering color, pose, or an attribute), the resulting image often undergoes unintended global structural changes such as composition shifts, pose alterations, or background reconstruction. Existing “spatial‑domain” approaches try to preserve structure by copying intermediate feature maps or attention maps from the generation guided by the original prompt into the generation guided by the edited prompt. However, these methods rely on empirical selection of which layers or maps to transfer, lack a solid theoretical foundation, and can be unstable.

The authors take a different perspective by analyzing the diffusion process in the frequency domain. They first note that natural images follow a power‑law spectral density (PSD ∝ ω⁻ᵝ), meaning most energy resides in low‑frequency components. In the latent diffusion model (LDM) pipeline, the VAE encoder compresses an image into a latent z₀ whose spectrum is similarly low‑frequency‑biased. During forward diffusion, Gaussian noise is added step‑by‑step, gradually amplifying high‑frequency components while low‑frequency energy diminishes. Conversely, during the reverse denoising process, low‑frequency components are recovered first, establishing the coarse structure (object layout, pose, overall composition). As denoising proceeds, higher frequencies become dominant, adding fine‑grained textures, colors, and details. The authors validate this hierarchy through PSD measurements at multiple timesteps and by conducting ablation experiments that block either low or high frequencies.

Based on this insight, they propose a Training‑Free Frequency Modulation Method (FMM). For a given timestep t, the noisy latent produced by the original prompt (zₜ⁰) and that produced by the edited prompt (zₜ¹) are transformed into the frequency domain via FFT. A frequency‑dependent weighting function

 wₜ(ω) = αₜ · exp(−β·ω) · γ(t)

is applied, where:

• αₜ controls overall scaling,
• β determines how quickly the weight decays with frequency,
• γ(t) is a dynamic decay factor that starts near 1 (strongly preserving low‑frequency information) and gradually decreases, allowing higher frequencies to follow the edited prompt.

The modulated latent is then reconstructed by inverse FFT:

 zₜ^mod = IFFT( wₜ·FFT(zₜ⁰) + (1 − wₜ)·FFT(zₜ¹) ).

This operation directly manipulates the latent noise before it is fed to the denoising network, thereby avoiding any need to select or inject internal feature maps. The dynamic decay ensures that early denoising steps keep the original structure intact, while later steps permit the edited prompt to influence fine details.

Key advantages:

1. No additional training or optimization – the method works with any pretrained LDM.
2. Global structural consistency – low‑frequency preservation prevents unwanted composition changes.
3. Fine‑grained semantic control – high‑frequency freedom enables precise attribute edits (color, texture, small object changes).
4. Computationally lightweight – FFT/IFFT are cheap; the method adds only a small overhead compared with spatial‑domain interventions.

The authors evaluate FMM on two public benchmarks for text‑guided image editing (e.g., PromptBench and ImageEditBench). They compare against state‑of‑the‑art spatial methods such as P2P, Pix2Pix‑Zero, TtfDf, and MasaCtrl. Metrics include:

• Structural fidelity: LPIPS and SSIM.
• Semantic alignment: CLIP‑Score and text‑image retrieval accuracy.
• Human preference: user studies rating structure preservation vs. semantic accuracy.

Results show that FMM reduces LPIPS by ~12% and improves SSIM by ~0.04 on average, indicating markedly better structure preservation. CLIP‑Score improves by ~0.07, demonstrating stronger adherence to the edited prompt. In user studies, over 85 % of participants preferred images generated with FMM for tasks involving subtle attribute changes. Moreover, because the method only requires FFT operations, inference time is about 30 % faster than methods that copy and blend high‑dimensional feature maps.

Limitations and future work: The current implementation operates on latent space; extending it to direct pixel‑space high‑resolution editing would require additional up‑sampling strategies. The choice of β and the schedule γ(t) is currently hand‑crafted; learning an adaptive schedule per dataset or prompt could further improve robustness. The authors also suggest exploring hybrid schemes that combine frequency modulation with attention‑based guidance for even richer control.

In summary, the paper provides a novel, theoretically grounded analysis of how frequency components drive the hierarchical emergence of structure and texture in diffusion models, and leverages this insight to propose a simple yet effective training‑free frequency modulation technique. This approach resolves the instability and structure‑drift problems of prior spatial‑domain methods, achieving a superior balance between preserving global composition and enabling precise semantic edits in text‑driven image generation.