TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control

Current controllable diffusion models typically rely on fixed architectures that modify intermediate activations to inject guidance conditioned on a new modality. This approach uses a static conditioning strategy for a dynamic, multi-stage denoising process, limiting the model’s ability to adapt its response as the generation evolves from coarse structure to fine detail. We introduce TC-LoRA (Temporally Modulated Conditional LoRA), a new paradigm that enables dynamic, context-aware control by conditioning the model’s weights directly. Our framework uses a hypernetwork to generate LoRA adapters on-the-fly, tailoring weight modifications for the frozen backbone at each diffusion step based on time and the user’s condition. This mechanism enables the model to learn and execute an explicit, adaptive strategy for applying conditional guidance throughout the entire generation process. Through experiments on various data domains, we demonstrate that this dynamic, parametric control significantly enhances generative fidelity and adherence to spatial conditions compared to static, activation-based methods. TC-LoRA establishes an alternative approach in which the model’s conditioning strategy is modified through a deeper functional adaptation of its weights, allowing control to align with the dynamic demands of the task and generative stage.

💡 Research Summary

The paper “TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control” presents a novel framework that fundamentally rethinks how conditional control is applied in diffusion models. It addresses a key limitation in prevailing methods like ControlNet, which use a static network architecture to inject conditional guidance (e.g., from depth maps) by modifying intermediate activations. The authors argue that this static conditioning is suboptimal for the inherently dynamic, multi-stage denoising process, where the ideal strategy should evolve from establishing coarse structure early on to refining fine details later.

To overcome this, TC-LoRA introduces a paradigm shift: dynamic weight adaptation. Instead of feeding conditions into a fixed network, TC-LoRA conditions the model’s weights directly. The core of the method is a hypernetwork, which is the only trainable component. For each denoising step, this hypernetwork takes the current timestep t and the conditioning signal y (e.g., a depth map) as input and generates on-the-fly a unique set of Low-Rank Adaptation (LoRA) matrices—B(t,y) and A(t,y). These dynamically generated adapters are then added to the frozen weights W0 of a pre-trained base diffusion model (e.g., Cosmos-Predict1), effectively creating a context-specific weight update: W' = W0 + B(t,y)A(t,y). This mechanism allows the model’s computational function itself to adapt at each step, enabling it to learn and execute an explicit strategy tailored to the specific needs of that generative stage.

The authors provide a theoretical motivation, arguing that modifying weights offers greater expressive power than merely modulating activations within a fixed function. They position TC-LoRA as a more efficient and powerful alternative to methods that add large, static control networks or those that only scale static LoRA updates over time.

Experimental validation is conducted on conditional image generation using depth maps. TC-LoRA adapters are trained exclusively on the MS-COCO dataset and evaluated on two benchmarks: a curated OpenImages set and the diverse TransferBench. The baseline for comparison is Cosmos-Transfer1, which augments the same base model with a ControlNet-style architecture. Results show that TC-LoRA significantly outperforms the baseline on quantitative metrics for condition adherence (NMSE and scale-invariant MSE), despite having only 251M trainable parameters compared to the baseline’s 900M. Qualitative examples demonstrate TC-LoRA’s superior ability to preserve fine-grained spatial details from the condition, such as the precise pose of an animal or the silhouette of pedestrians, which the baseline method often fails to capture accurately.

In conclusion, TC-LoRA establishes a new approach for controllable generation by enabling deep, functional adaptation of a model’s weights in response to both the conditioning input and the progression of the generative process. This leads to enhanced fidelity, efficiency, and steerability. The paper suggests promising future directions, including extending the framework to text-to-video generation, where the hypernetwork could balance per-frame spatial conditioning with temporal consistency across frames.