Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits

We introduce a novel pipeline for joint audio-visual editing that enhances the coherence between edited video and its accompanying audio. Our approach first applies state-of-the-art video editing techniques to produce the target video, then performs audio editing to align with the visual changes. To achieve this, we present a new video-to-audio generation model that conditions on the source audio, target video, and a text prompt. We extend the model architecture to incorporate conditional audio input and propose a data augmentation strategy that improves training efficiency. Furthermore, our model dynamically adjusts the influence of the source audio based on the complexity of the edits, preserving the original audio structure where possible. Experimental results demonstrate that our method outperforms existing approaches in maintaining audio-visual alignment and content integrity.

💡 Research Summary

The paper introduces a two‑stage pipeline for coherent audio‑visual editing that first leverages state‑of‑the‑art video editing methods to produce a target video, and then edits the accompanying audio so that it aligns with the visual changes while preserving as much of the original audio structure as possible. The authors define a new task—audio editing following video edits—where the model must simultaneously satisfy three constraints: fidelity to a textual prompt, tight audio‑visual synchronization with the edited video, and preservation of the source audio’s temporal and structural characteristics (e.g., event timing, background continuity).

To address this task, the authors extend a recent video‑to‑audio generation model (MMAudio) by adding the source audio as an additional conditioning modality. They propose a hierarchical acoustic feature representation based on frame‑wise loudness computed over recursively split frequency bands. This representation yields multiple levels of detail, from coarse to fine, and can be selectively masked to control how much structural information is injected during generation. A novel data‑augmentation technique called “detail‑temporal masking” randomly masks entire detail levels or temporal segments of these acoustic features during training, encouraging the model to be robust to varying degrees of structure preservation.

The core generative engine remains a flow‑matching framework. In this setting, a neural network learns a time‑dependent velocity field that transforms a simple prior distribution into the target audio distribution. Conditional inputs (text prompt, video features, and the hierarchical acoustic features) are incorporated via classifier‑free guidance, allowing the guidance weight to balance prompt fidelity against video alignment. The model architecture retains the multi‑modal transformer and audio‑only DiT blocks of MMAudio, but introduces two key modifications: (1) modulation of the audio latent representations by the acoustic features (after linear interpolation and projection to match dimensionality), and (2) modulation of the Syncformer features—responsible for video‑audio synchronization—by the same acoustic features on a frame‑wise basis. Both modulation pathways are initialized to act as identity functions; they remain frozen for the first half of training and are later fine‑tuned, preventing the model from over‑relying on acoustic conditioning at the expense of textual and visual cues.

An adaptive conditioning mechanism is also proposed. The authors compute an “editability score” that quantifies the semantic distance between the source audio and the target video (e.g., using CLIP‑based similarity). This score determines how aggressively the acoustic features are masked: high scores (minor edits) keep more of the source structure, while low scores (major edits) suppress detailed acoustic conditioning, allowing the model to generate more novel content.

Experiments are conducted on video‑audio datasets at 20 fps, a substantial increase over prior joint editing works limited to 1–4 fps. Quantitative metrics include Audio‑Visual Alignment (AVA) and Structure Preservation (SP), both of which show significant improvements over baselines that either edit modalities independently or use low‑frame‑rate joint models. Human evaluations corroborate these findings, indicating higher prompt fidelity, smoother temporal dynamics, and better background ambience continuity. Qualitative examples demonstrate the model’s ability to retain original event timing while inserting new sounds (e.g., adding seagull chirps after a scene cut) and to adapt to complex visual edits such as background replacements.

In summary, the paper presents a comprehensive solution for high‑frame‑rate, coherent audio‑visual editing by (1) decoupling video and audio editing into sequential stages, (2) enriching a flow‑matching audio generator with hierarchical acoustic conditioning, (3) employing detail‑temporal masking for efficient training, and (4) dynamically adjusting conditioning strength based on an editability score. The approach outperforms existing methods in both objective and subjective evaluations, opening the door to more sophisticated multimedia editing workflows where audio and video remain tightly synchronized even after substantial visual modifications. Future work may explore richer acoustic descriptors, larger multimodal datasets, and real‑time inference for interactive editing applications.