Detecting and Mitigating Memorization in Diffusion Models through Anisotropy of the Log-Probability

Diffusion-based image generative models produce high-fidelity images through iterative denoising but remain vulnerable to memorization, where they unintentionally reproduce exact copies or parts of training images. Recent memorization detection methods are primarily based on the norm of score difference as indicators of memorization. We prove that such norm-based metrics are mainly effective under the assumption of isotropic log-probability distributions, which generally holds at high or medium noise levels. In contrast, analyzing the anisotropic regime reveals that memorized samples exhibit strong angular alignment between the guidance vector and unconditional scores in the low-noise setting. Through these insights, we develop a memorization detection metric by integrating isotropic norm and anisotropic alignment. Our detection metric can be computed directly on pure noise inputs via two conditional and unconditional forward passes, eliminating the need for costly denoising steps. Detection experiments on Stable Diffusion v1.4 and v2 show that our metric outperforms existing denoising-free detection methods while being at least approximately 5x faster than the previous best approach. Finally, we demonstrate the effectiveness of our approach by utilizing a mitigation strategy that adapts memorized prompts based on our developed metric. The code is available at https://github.com/rohanasthana/memorization-anisotropy .

💡 Research Summary

Diffusion models have become the dominant class of generative models for images, videos, and other modalities, yet they remain prone to memorization—the unintended reproduction of training data either exactly or with minor variations. Existing detection methods largely rely on the norm of the score difference between conditional and unconditional models (‖sΔ‖). While effective at medium‑to‑high noise levels, these norm‑based metrics assume that the underlying log‑probability distribution is isotropic, i.e., curvature is the same in every direction. The authors demonstrate that this assumption breaks down in the low‑noise regime, where the diffusion process approaches the data manifold and the log‑probability becomes anisotropic: curvature varies strongly across directions, and memorized samples manifest as a sharp peak along a specific direction.

In the anisotropic regime, the authors discover that memorized prompts produce a strong angular alignment between the guidance vector (the difference between conditional and unconditional scores) and the unconditional score itself. In other words, the guidance points almost exactly in the direction where the unconditional score is already large, indicating that the model’s probability mass is concentrated along the memorized image’s manifold direction.

To capture both regimes, the paper proposes a composite detection metric M:

1. Isotropic term – the norm ‖sΔ‖ evaluated at a relatively high timestep t_high, which measures overall curvature when the distribution is near‑isotropic.
2. Anisotropic term – the cosine similarity between the guidance vector g = sθ(x_t, c) − sθ(x_t, ∅) and the unconditional score sθ(x_t, ∅) evaluated at a low timestep t_low, which captures directional alignment in the anisotropic regime.

M = α·‖sΔ‖{t_high} + β·cos(g, sθ){t_low}, with α and β tuned on a validation set. Crucially, both terms can be computed with only two forward passes (conditional and unconditional) on pure noise, eliminating any costly denoising trajectory.

The authors evaluate the metric on Stable Diffusion v1.4 and v2.0 using the MemBench benchmark, which contains known memorized image‑prompt pairs. Compared to prior denoising‑free methods (e.g., WEN‑2024, JEON‑2025) and a recent curvature‑tracking approach (BROKMAN‑2025), the composite metric achieves higher AUC/AP scores (improvements of 4–7 %) while being ≈5× faster (≈0.12 s per sample on an A100 GPU). Ablation studies confirm that removing either the isotropic or anisotropic component degrades performance, underscoring the necessity of handling both regimes.

Beyond detection, the paper introduces a mitigation strategy that leverages the same metric. When M exceeds a threshold, the prompt is augmented: the classifier‑free guidance weight w is reduced, small Gaussian noise is added to the text embedding, and optionally the random seed is resampled. These modifications preserve the semantic content of the prompt (CLIP text‑image similarity drops only slightly) but disrupt the precise directional alignment that leads to memorization. In experiments, this reduces image‑level similarity to the original training image (SSIM from 0.31 to 0.07) while maintaining high aesthetic quality and text‑image alignment.

The paper’s contributions are threefold:

• A theoretical analysis showing that norm‑based memorization detection is only valid under isotropic log‑probability, and that anisotropy emerges in low‑noise diffusion steps.
• A novel, denoising‑free detection metric that combines curvature (norm) and angular alignment, delivering superior accuracy and speed.
• An inference‑time mitigation technique that uses the metric to adapt prompts on the fly, effectively preventing memorized outputs.

Overall, this work bridges a gap between the geometric understanding of diffusion trajectories and practical tools for privacy‑preserving generative AI. By exploiting both isotropic and anisotropic characteristics of the log‑probability, it offers a scalable solution for detecting and mitigating memorization in large‑scale diffusion models, a crucial step toward responsible deployment.