ADx3: A Collaborative Workflow for High-Quality Accessible Audio Description

Audio description (AD) makes video content accessible to blind and low-vision (BLV) audiences, but producing high-quality descriptions is resource-intensive. Automated AD offers scalability, and prior studies show human-in-the-loop editing and user queries effectively improve narration. We introduce ADx3, a novel framework integrating these three modules: GenAD, upgrading baseline description generation with modern vision-language models (VLMs) guided by accessibility-informed prompting; RefineAD, supporting BLV and sighted users to view and edit drafts through an inclusive interface; and AdaptAD, enabling on-demand user queries. We evaluated GenAD in a study where seven accessibility specialists reviewed VLM-generated descriptions using professional guidelines. Findings show that with tailored prompting, VLMs produce good descriptions meeting basic standards, but excellent descriptions require human edits (RefineAD) and interaction (AdaptAD). ADx3 demonstrates collaborative workflows for accessible content creation, where components reinforce one another and enable continuous improvement: edits guide future baselines and user queries reveal gaps in AI-generated and human-authored descriptions.

💡 Research Summary

The paper introduces ADx3, a comprehensive workflow that unifies three essential components for producing high‑quality audio description (AD) for blind and low‑vision (BLV) audiences: (1) GenAD, an automated baseline generation module that leverages state‑of‑the‑art vision‑language models (VLMs); (2) RefineAD, an inclusive editing interface that allows both BLV users and sighted editors to view, correct, and enrich AI‑generated drafts; and (3) AdaptAD, an on‑demand query system that lets end‑users request additional details while watching a video.

GenAD upgrades prior caption‑based pipelines by employing three leading VLMs—Qwen2.5‑VL (open‑source), Gemini 1.5 Pro (Google DeepMind), and GPT‑4o (OpenAI). The authors download videos via yt‑dlp, extract frames with ffmpeg, and segment scenes using OpenCLIP embeddings and cosine‑similarity thresholds. For each scene, a contextual prompt—crafted iteratively with input from an accessibility consultant—guides the VLM to produce a concise yet comprehensive description covering objects, characters, actions, on‑screen text, and environmental cues. The prompt explicitly encodes accessibility guidelines (DCMP, NCAM, WCAG 2.0 AA) to encourage consistent style, timing, and level of detail.

RefineAD provides a web‑based authoring environment that integrates a synchronized timeline, audio playback, and screen‑reader compatible controls. Editors can edit the generated text, adjust pause durations, merge or split descriptions, and add speaker tags. The interface also visualizes the original video frames, enabling BLV editors to verify visual details through audio cues or tactile feedback devices. This module bridges the gap between raw AI output and professional AD standards, reducing the cognitive load on volunteers and allowing rapid iteration.

AdaptAD empowers the final BLV viewer to interact with the description in real time. Users can pose natural‑language questions such as “What is the sign saying?” or “Who just entered the room?” The system routes the query to the same VLM used in GenAD, which re‑examines the relevant scene and returns a targeted supplemental narration. All queries are logged; the authors argue that recurring question patterns (e.g., frequent requests for character identity or on‑screen text) expose systematic blind spots in the AI baseline and human edits. These insights feed back into future prompt engineering and fine‑tuning of the VLMs, creating a closed‑loop improvement cycle.

To evaluate the framework, the authors conducted a workshop with seven accessibility specialists. Participants reviewed anonymized descriptions generated by each of the three VLMs and rated them against professional AD guidelines. Results showed that, with accessibility‑informed prompting, all models consistently produced “good” drafts that met basic standards (accurate object identification, reasonable fluency). However, achieving “excellent” quality—characterized by precise timing, nuanced emotional tone, and avoidance of hallucinations—required substantial human refinement via RefineAD and supplemental information supplied through AdaptAD. The study highlighted specific weaknesses: (i) difficulty handling text‑heavy scenes, (ii) occasional misidentification of characters in crowded shots, and (iii) limited temporal coherence across scene boundaries.

The paper’s contributions are twofold: (1) a systematic evaluation of three cutting‑edge VLMs for AD baseline generation, identifying strengths (rich visual detail, integrated OCR) and limitations (temporal reasoning, hallucination risk); and (2) the design of a unified, scalable pipeline that embeds human‑in‑the‑loop editing and user‑driven interactivity, thereby reducing the labor required for high‑quality AD while preserving the flexibility to tailor descriptions to individual viewing goals.

Limitations acknowledged include the latency introduced by real‑time query processing, and the current VLMs’ insufficient modeling of long‑range temporal dependencies, which can cause inconsistencies in multi‑scene narratives. Future work is proposed to incorporate multimodal transformer architectures with explicit temporal attention, to explore pre‑fetching strategies for query responses, and to conduct longitudinal user studies with BLV participants to measure comprehension, satisfaction, and workflow efficiency.

In sum, ADx3 demonstrates that a collaborative workflow—combining advanced AI generation, accessible human editing, and interactive user queries—can substantially improve the scalability and quality of audio description, moving the field closer to universal video accessibility.