Contextualized Visual Personalization in Vision-Language Models

Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user’s specific experiences, as they lack the ability to associate visual inputs with a user’s accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.

💡 Research Summary

The paper introduces a new research problem called “contextualized visual personalization,” which refers to the ability of vision‑language models (VLMs) to generate responses that are personalized based on a user’s accumulated visual‑textual history. Existing VLMs excel at image recognition but lack mechanisms to link new visual inputs with long‑term, user‑specific context, leading to generic outputs that ignore personal experiences.

To address this gap, the authors propose CoViP (Contextualized Visual Personalization via Image Captioning), a unified framework that treats personalized image captioning as the core proxy task for learning the underlying personalization process. The key idea is to decompose the model into two components: (1) a contextual visual encoder hθ that ingests the current image and the user’s multimodal history to produce a personalized latent representation z, and (2) a task‑specific generator gθ that produces the final textual response conditioned on z and the user prompt. By focusing training on hθ, the framework can improve personalization across a wide range of downstream tasks.

A dedicated benchmark is built to evaluate this capability. Using a generative image model, the authors synthesize 2.8 K training and 1.3 K test samples that contain 1–4 visual concepts (people, objects, animals) per image, together with multi‑turn dialogues grounded in factual details (locations, timestamps, events). Positive and negative examples are interleaved to force the model to both recognize visual concepts and correctly retrieve the relevant personal context. Automatic quality filtering with a separate text‑generation VLM ensures that each image matches its prompt and that the visual content is faithful.

Training proceeds with reinforcement‑learning‑based post‑training (RL‑Post‑Training). Starting from a pretrained VLM, the model is fine‑tuned on the personalized caption benchmark using a reward that combines standard caption metrics (BLEU, CIDEr) with a personalization accuracy term that checks whether the generated caption correctly references the user‑specific concepts. This RL step is shown to be more effective than conventional supervised fine‑tuning, as it explicitly encourages the model to align visual perception with personal memory.

At inference time, CoViP employs Caption‑Augmented Generation (CAG). The model first generates a personalized caption for the query image; this caption is then fed back as an additional conditioning signal for the final response. CAG leverages the fine‑grained details captured in the caption, leading to richer and more consistent personalized outputs.

To verify that improvements are not due to “textual shortcuts” (e.g., retrieving answers directly from the context without visual understanding), the authors design a suite of diagnostic tasks. One set of diagnostics removes visual cues, testing whether the model can still answer correctly; another mixes positive and negative context items, requiring the model to correctly select the relevant personal information. Results show that many open‑source and proprietary VLMs rely heavily on textual hints and exhibit unstable performance on these diagnostics, whereas CoViP consistently achieves high accuracy, confirming genuine visual‑context integration.

Experimental results demonstrate substantial gains: CoViP improves caption BLEU‑4 by an average of 7.3 points and CIDEr by 12.5 points over strong baselines, and it yields 10–15 % relative improvements on downstream tasks such as contextual question answering, name recall, and proactive personalized dialogue. Ablation studies reveal that the RL post‑training stage is crucial—removing it leads to a dramatic drop in both caption quality and diagnostic accuracy. Adding CAG further boosts performance by 3–5 % across metrics.

The paper’s contributions are fourfold: (1) formalizing contextualized visual personalization and proposing a rigorous evaluation protocol, (2) introducing CoViP, a framework that unifies personalized captioning, RL‑based post‑training, and caption‑augmented generation, (3) creating diagnostic benchmarks that explicitly rule out textual shortcuts, and (4) providing extensive empirical evidence that CoViP yields robust, generalizable personalization across a variety of VLMs.

Limitations include reliance on synthetic data; real‑world user logs have not yet been tested, and the RL reward design may be sensitive to metric weighting. Future work could involve large‑scale user‑driven evaluations, integration with external multimodal memory modules, and exploration of more sophisticated reward shaping to further close the gap between synthetic benchmarks and real‑world personalization needs.