BLENDER: Blended Text Embeddings and Diffusion Residuals for Intra-Class Image Synthesis in Deep Metric Learning

The rise of Deep Generative Models (DGM) has enabled the generation of high-quality synthetic data. When used to augment authentic data in Deep Metric Learning (DML), these synthetic samples enhance intra-class diversity and improve the performance of downstream DML tasks. We introduce BLenDeR, a diffusion sampling method designed to increase intra-class diversity for DML in a controllable way by leveraging set-theory inspired union and intersection operations on denoising residuals. The union operation encourages any attribute present across multiple prompts, while the intersection extracts the common direction through a principal component surrogate. These operations enable controlled synthesis of diverse attribute combinations within each class, addressing key limitations of existing generative approaches. Experiments on standard DML benchmarks demonstrate that BLenDeR consistently outperforms state-of-the-art baselines across multiple datasets and backbones. Specifically, BLenDeR achieves 3.7% increase in Recall@1 on CUB-200 and a 1.8% increase on Cars-196, compared to state-of-the-art baselines under standard experimental settings.

💡 Research Summary

The paper introduces BLENDER (Blended Text Embeddings and Diffusion Residuals), a novel diffusion‑based data‑augmentation technique specifically designed for Deep Metric Learning (DML). DML aims to learn an embedding space where samples from the same class are close while samples from different classes are far apart. Performance of DML models heavily depends on the intra‑class diversity of the training set; limited pose, background, or lighting variations can cause over‑fitting and poor generalisation. Existing augmentation methods for DML, such as CutMix, DDIM inversion, or image‑to‑image diffusion, either produce unrealistic artefacts or suffer from noisy labels, making them sub‑optimal for metric‑learning objectives.

BLENDER tackles these issues through two complementary mechanisms: Text Embedding Interpolation (TEI) and Residual Set Operations (RSO).

1. Text Embedding Interpolation (TEI) – The method defines three types of prompts: a target anchor prompt (c₁) containing the class token and the desired novel attribute, an attribute‑donor prompt (c₂) that uses a different but related class known to co‑occur with the same attribute, and a set of context‑prior prompts (c₃…cₙ) that provide paraphrases or semantically related attributes. During the early diffusion steps a cosine‑ramped weight γ(t) interpolates between the embeddings of c₁ and c₂ (α₁(t)=1−γ(t), α₂(t)=γ(t)), producing a mixed embedding h_mix(t). This early injection steers the latent toward the target attribute while preserving the class identity.

2. Residual Set Operations (RSO) – At each denoising timestep t the diffusion U‑Net predicts noise residuals ε_i = ε_θ(x_t, t, E(c_i)) for each prompt. The residual relative to the mixed embedding is r_i = ε_i − ε_mix. Two set‑theoretic operators are then applied:

   • Union (R∪) – For a selected subset I∪ of prompts that should contribute the attribute, the normalized residuals are averaged:
     R∪ = Σ_{i∈I∪} r_i / (‖r_i‖₂ + δ).
     This operation amplifies any attribute present in at least one prompt, making the method robust to varied phrasing.

   • Intersection (R∩) – For a subset I∩, the residuals are stacked into a matrix M, and the first principal component v₁ is extracted via singular‑value decomposition. The mean residual μ is projected onto v₁, yielding
     R∩ = ⟨μ, v₁⟩ v₁.
     This captures the common direction shared by all prompts, ensuring coherent attribute synthesis.

   Time‑varying weights β∪(t) and β∩(t) control the influence of the two operators throughout the diffusion trajectory. The combined residual is added to the standard classifier‑free guidance term r_cfg = ε_mix − ε_∅, after orthogonalising it to avoid over‑steering. The final noise update becomes:
   b_ε(t) = ε_∅ + w_cfg(t)·r_cfg + β∪(t)·R∪ + β∩(t)·R∩.

By integrating TEI and RSO, BLENDER can generate images that (i) exhibit novel attribute combinations (different poses, backgrounds, lighting, etc.) within a given class, and (ii) retain the semantic core of the class, thereby providing clean labels for DML training.

Experimental validation – The authors fine‑tune a Stable Diffusion model with LoRA on each class of two standard DML benchmarks: CUB‑200‑2011 (200 bird species) and Cars‑196 (196 car models). Using BLENDER, they synthesize additional training images with controlled attribute variations. They evaluate several backbone architectures (ResNet‑50, ViT‑Base, EfficientNet‑B3, and a recent transformer‑based encoder) under common DML losses (triplet, proxy‑NCA, and hyperbolic losses). Across all settings, BLENDER consistently outperforms prior augmentation baselines, achieving a 3.7 % absolute gain in Recall@1 on CUB‑200 and a 1.8 % gain on Cars‑196 compared to the previous state‑of‑the‑art. Additional metrics such as Recall@5, NMI, and MAP also show improvements. Ablation studies demonstrate that the Union operation is crucial when multiple paraphrased prompts are used, while the Intersection operation preserves attribute consistency when prompts share a common semantic core.

Limitations and future work – The residual set operations involve high‑dimensional vector arithmetic and an SVD step, which can become computationally intensive for large prompt sets. The current pipeline is tightly coupled to text‑to‑image diffusion models; extending it to GANs or VAE‑based generators would require redesigning the residual formulation. Moreover, the selection of attribute donor and context prompts relies on manual design; automating attribute discovery could further broaden applicability.

Conclusion – BLENDER provides a mathematically grounded, controllable approach to enrich intra‑class diversity for metric‑learning datasets. By operating directly on diffusion residuals and leveraging set‑theoretic composition, it overcomes the trade‑off between attribute variety and label fidelity that has limited previous generative augmentation methods. The demonstrated performance gains across multiple datasets and backbones suggest that BLENDER could become a standard component in DML pipelines and may inspire similar residual‑based augmentation strategies in other representation‑learning domains.