HERS: Hidden-Pattern Expert Learning for Risk-Specific Vehicle Damage Adaptation in Diffusion Models

Recent advances in text-to-image (T2I) diffusion models have enabled increasingly realistic synthesis of vehicle damage, raising concerns about their reliability in automated insurance workflows. The ability to generate crash-like imagery challenges the boundary between authentic and synthetic data, introducing new risks of misuse in fraud or claim manipulation. To address these issues, we propose HERS (Hidden-Pattern Expert Learning for Risk-Specific Damage Adaptation), a framework designed to improve fidelity, controllability, and domain alignment of diffusion-generated damage images. HERS fine-tunes a base diffusion model via domain-specific expert adaptation without requiring manual annotation. Using self-supervised image-text pairs automatically generated by a large language model and T2I pipeline, HERS models each damage category, such as dents, scratches, broken lights, or cracked paint, as a separate expert. These experts are later integrated into a unified multi-damage model that balances specialization with generalization. We evaluate HERS across four diffusion backbones and observe consistent improvements: plus 5.5 percent in text faithfulness and plus 2.3 percent in human preference ratings compared to baselines. Beyond image fidelity, we discuss implications for fraud detection, auditability, and safe deployment of generative models in high-stakes domains. Our findings highlight both the opportunities and risks of domain-specific diffusion, underscoring the importance of trustworthy generation in safety-critical applications such as auto insurance.

💡 Research Summary

The paper addresses a critical gap in the deployment of text‑to‑image (T2I) diffusion models for automobile insurance workflows: while modern diffusion models can generate photorealistic pictures, they often fail to reproduce the fine‑grained damage patterns (dents, scratches, cracked lights, etc.) that are essential for liability assessment, fraud detection, and automated claim processing. To bridge this gap, the authors propose HERS (Hidden‑Pattern Expert Learning for Risk‑Specific Damage Adaptation), a fully automated framework that creates domain‑specific “experts” without any human‑annotated data and then merges them into a single, multi‑damage diffusion model.

Core methodology

1. Prompt synthesis – A large language model (GPT‑4) is seeded with a few exemplar prompts for each damage category (dent, scrape, torn bumper, cracked paint, broken light). It generates a large, diverse set of prompts, which are filtered by ROUGE‑L similarity to keep only semantically distinct items. This yields a structured prompt bank P covering typical parts, narrative contexts, and deliberately implausible scenarios.
2. Synthetic image generation – Each prompt in P is fed to a pretrained T2I backbone (e.g., Stable Diffusion XL, VQ‑Diffusion, Versatile Diffusion, MoLE). The resulting (prompt, image) pairs form a self‑supervised dataset D that contains both realistic accident depictions and adversarially crafted “fraud‑like” scenes.
3. Domain‑specific LoRA experts – For each domain t (e.g., typical parts, scene narratives, implausible scenarios) a lightweight Low‑Rank Adaptation (LoRA) module is trained. The LoRA updates ΔWₜ = BₜAₜ have a small rank r (≈128) compared to the full weight matrix, allowing efficient specialization to subtle visual cues such as hairline cracks or asymmetric paint chipping.
4. Weight‑space merging – All LoRA experts are merged by simple arithmetic averaging of their A and B matrices, producing a consolidated weight set W* = W₀ + B* A*. The merged model can generate any of the learned damage types in a zero‑shot fashion, eliminating the need for inference‑time routing or external classifiers.

Evaluation protocol
The authors evaluate HERS on a proprietary 2‑million‑entry benchmark built in collaboration with an insurance startup. The benchmark contains structured textual descriptions (damage type, part location, accident context) paired with real vehicle images. Two complementary metrics are used:

• Semantic alignment – A VQA‑based protocol where a large language model creates targeted questions about the generated image; a pretrained VQA model answers them, and accuracy serves as a proxy for text‑image fidelity.
• Human‑aligned quality – Preference‑based reward models (PickScore, ImageReward, HPS) derived from large‑scale human preference data are applied to assess realism, relevance, and overall visual quality.

Results
Across four diffusion backbones (VQ‑Diffusion, Versatile Diffusion, SDXL, MoLE), HERS consistently outperforms baselines. On the “Car Insurance” prompt set, HERS achieves an average Human Preference Score (HPS) of 53.4 % versus 48.2 % for the strongest baseline (MoLE) and improves VQA‑based text fidelity by +5.5 percentage points. Similar gains are observed on a “Car Garage” prompt set. Qualitative examples show that HERS preserves fine‑grained damage details (e.g., a subtle dent on a hood combined with a cracked headlight) where baseline models either blur or omit one of the elements. Moreover, the inclusion of implausible scenarios demonstrates that HERS can generate controlled “fraud‑like” images, a capability useful for stress‑testing detection pipelines.

Limitations and future work
The approach relies on the quality of automatically generated prompts and synthetic images; any bias or systematic error in the LLM or base diffusion model propagates to the experts. Simple averaging of LoRA weights may cause interference when damage categories overlap (e.g., a dent that also creates a paint chip). The authors suggest exploring adaptive weighting or routing mechanisms, as well as cross‑validation with real claim images to close the domain gap.

Broader impact
HERS offers a cost‑effective way to produce high‑fidelity, risk‑aware synthetic data for insurance applications. Potential benefits include: (1) augmenting scarce real‑world accident images for training more robust claim‑assessment models; (2) generating controlled fraudulent examples to improve detection systems; (3) providing a testbed for policy makers to evaluate the ethical implications of generative AI in safety‑critical domains. At the same time, the paper acknowledges the dual‑use nature of the technology and calls for responsible deployment guidelines.

In summary, HERS introduces a novel self‑supervised pipeline that automatically creates damage‑specific LoRA experts, merges them into a unified diffusion model, and demonstrates measurable improvements in both semantic alignment and human‑perceived quality. The work advances the state of the art in domain‑specific generative modeling and opens new research directions for trustworthy AI in high‑stakes industries such as auto insurance.