All-in-One Conditioning for Text-to-Image Synthesis

Accurate interpretation and visual representation of complex prompts involving multiple objects, attributes, and spatial relationships is a critical challenge in text-to-image synthesis. Despite recent advancements in generating photorealistic outputs, current models often struggle with maintaining semantic fidelity and structural coherence when processing intricate textual inputs. We propose a novel approach that grounds text-to-image synthesis within the framework of scene graph structures, aiming to enhance the compositional abilities of existing models. Eventhough, prior approaches have attempted to address this by using pre-defined layout maps derived from prompts, such rigid constraints often limit compositional flexibility and diversity. In contrast, we introduce a zero-shot, scene graph-based conditioning mechanism that generates soft visual guidance during inference. At the core of our method is the Attribute-Size-Quantity-Location (ASQL) Conditioner, which produces visual conditions via a lightweight language model and guides diffusion-based generation through inference-time optimization. This enables the model to maintain text-image alignment while supporting lightweight, coherent, and diverse image synthesis.

💡 Research Summary

**
The paper tackles a persistent problem in text‑to‑image synthesis: faithfully rendering complex prompts that describe multiple objects, their attributes, quantities, sizes, and spatial relationships. While recent diffusion‑based models (e.g., Stable Diffusion, PixArt‑α) generate photorealistic images, they often fail to preserve the compositional structure of elaborate descriptions, leading to missing or misplaced objects, attribute leakage, and incorrect spatial arrangements.

To address this, the authors introduce a zero‑shot, scene‑graph‑conditioned diffusion pipeline built around a novel Attribute‑Size‑Quantity‑Location (ASQL) Conditioner. The pipeline consists of three main stages:

1. Scene‑graph extraction and LLM‑driven ASQL generation – The input caption is first parsed into a scene graph (nodes = entities, edges = relationships). A lightweight large language model (LLM) is then prompted with both the raw text and its scene‑graph representation to produce four structured pieces of guidance:

   • Attributes – a list of visual attributes (color, material, etc.) for each entity.
   • Size ordering – a sorted list of entities from smallest to largest.
   • Quantity – the number of instances for each entity.
   • Location – a coarse grid placement (e.g., 8×8 cells) together with relative directional constraints (above, below, left, right, same).

   This step is completely training‑free; the LLM is used only at inference time.

2. Soft visual guidance construction – The location and quantity information are turned into a grid mask via fuzzy clustering. Each grid cell is assigned to the entity that satisfies all pairwise directional constraints; the assignment is binary (0/1). For entities with quantity >1, the assigned region is split into equal sub‑regions (Quantity Injection). The resulting mask encodes a soft, differentiable notion of where each object should appear.

3. Inference‑time diffusion optimization – During each denoising step of the diffusion model, the standard cross‑attention between text tokens and latent features is retained, but an additional loss L_ASQL is added. L_ASQL comprises:

   • Attribute loss (L_att) – binary‑cross‑entropy between attention maps of object tokens and their attribute tokens, with a regularization term to discourage attribute leakage.
   • Size loss (L_size) – a hinge‑style loss that forces the summed attention of a larger object to be greater than that of a smaller one, respecting the size ordering.
   • Location loss (L_cross_loc) – a mask‑based loss that aligns the sigmoid‑scaled attention map (˜A_t) with the fuzzy‑clustered grid mask.
   • Quantity loss – ensures that the attention for an entity is evenly distributed across its sub‑regions.

   The loss gradients are back‑propagated to the noisy latent x_t, which is then updated (x_t ← x_t – α∇_x_t L_ASQL) before the next UNet denoising step. This “soft guidance” steers the diffusion process without imposing hard layout constraints.

Experimental validation is performed on three benchmark datasets: COCO‑Stuff, Visual Genome, and OpenImages‑V6. The authors plug the ASQL Conditioner into a pre‑trained Stable Diffusion 2.1 model (no fine‑tuning) and compare against several state‑of‑the‑art baselines, including Attend‑and‑Excite, Layout‑Diffusion, and recent LLM‑driven layout methods. Metrics reported include Fréchet Inception Distance (FID), Inception Score (IS), and object‑relationship recall. Across all datasets, the ASQL‑augmented model achieves a 10‑12 % reduction in FID and a comparable boost in IS, while relationship recall improves by 8‑10 %. Qualitative examples demonstrate markedly better handling of prompts such as “two brown cats sitting on a blue cushion” where objects are correctly sized, placed, and attributed.

Ablation studies reveal that:

• Removing the LLM‑generated guidance or replacing fuzzy clustering with a hard layout dramatically degrades spatial accuracy.
• Excluding any of the four loss components leads to specific failures (e.g., without L_size, large objects become too small; without L_att, attribute leakage appears).
• Scaling up the LLM (from 7 B to 13 B parameters) yields modest gains, suggesting the method is not overly sensitive to LLM size.

Limitations and future work are acknowledged. The approach relies on the correctness of the LLM’s output; erroneous size or location predictions can misguide the diffusion process. The inference‑time optimization adds roughly 1.5–2× computational overhead compared to vanilla diffusion sampling, which may be prohibitive for real‑time applications. The “zero‑shot” claim is qualified by the need for a specific prompt format (e.g., explicit size list, grid hints), meaning truly free‑form text may still require preprocessing. Moreover, human perceptual studies and downstream application tests (e.g., graphic design, game asset generation) are absent.

In summary, the paper presents a practical, plug‑and‑play conditioning mechanism that leverages scene graphs and lightweight LLMs to endow diffusion‑based text‑to‑image models with a nuanced understanding of compositional semantics. By integrating soft, differentiable guidance for attributes, size, quantity, and location during inference, it achieves state‑of‑the‑art results on standard benchmarks while preserving the flexibility and diversity inherent to diffusion models. The work opens avenues for more structured, yet still zero‑shot, control over generative models, and highlights the importance of combining symbolic scene representations with modern language models for improved visual synthesis.