From Ideal to Real: Stable Video Object Removal under Imperfect Conditions

Removing objects from videos remains difficult in the presence of real-world imperfections such as shadows, abrupt motion, and defective masks. Existing diffusion-based video inpainting models often struggle to maintain temporal stability and visual consistency under these challenges. We propose Stable Video Object Removal (SVOR), a robust framework that achieves shadow-free, flicker-free, and mask-defect-tolerant removal through three key designs: (1) Mask Union for Stable Erasure (MUSE), a windowed union strategy applied during temporal mask downsampling to preserve all target regions observed within each window, effectively handling abrupt motion and reducing missed removals; (2) Denoising-Aware Segmentation (DA-Seg), a lightweight segmentation head on a decoupled side branch equipped with Denoising-Aware AdaLN and trained with mask degradation to provide an internal diffusion-aware localization prior without affecting content generation; and (3) Curriculum Two-Stage Training: where Stage I performs self-supervised pretraining on unpaired real-background videos with online random masks to learn realistic background and temporal priors, and Stage II refines on synthetic pairs using mask degradation and side-effect-weighted losses, jointly removing objects and their associated shadows/reflections while improving cross-domain robustness. Extensive experiments show that SVOR attains new state-of-the-art results across multiple datasets and degraded-mask benchmarks, advancing video object removal from ideal settings toward real-world applications.

💡 Research Summary

The paper tackles the long‑standing challenge of video object removal (VOR) under realistic imperfections such as shadows, reflections, abrupt object motion, and defective segmentation masks. While recent diffusion‑based video inpainting models achieve impressive visual quality when supplied with perfect masks and well‑aligned frames, they break down in real‑world scenarios where masks are noisy, temporally sparse, or mis‑aligned due to fast motion. The authors propose Stable Video Object Removal (SVOR), a three‑pronged framework designed explicitly to survive these imperfections.

1. Mask Union for Stable Erasure (MUSE).
   Conventional pipelines down‑sample masks temporally to reduce computational load. When an object moves quickly, its presence may be captured only in a few frames; down‑sampling discards those frames, leading to “under‑erasure” and flickering. MUSE addresses this by performing a logical OR over all masks inside a sliding temporal window (e.g., five frames) before down‑sampling. The union retains every location the object occupies within the window, guaranteeing that transient appearances are never lost. This operation adds negligible overhead and dramatically reduces missed erasures in high‑motion sequences.

2. Denoising‑Aware Segmentation (DA‑Seg).
   Even with MUSE, imperfect masks can still mis‑localize the target region. DA‑Seg introduces a lightweight side‑branch segmentation head that predicts a soft mask conditioned on the diffusion timestep via Denoising‑Aware AdaLN. By feeding the current timestep embedding into the normalization parameters, the segmentation adapts from coarse to fine as the diffusion process proceeds, remaining robust under high‑noise steps. Crucially, the predicted mask is never fed back into the main DiT backbone; it is only used for an auxiliary loss against the down‑sampled ground‑truth mask. This decoupling preserves the generative capacity of the backbone while providing an internal, diffusion‑aware localization prior that stabilizes erasure when external masks are degraded.

3. Curriculum Two‑Stage Training.
   Stage I pre‑trains the backbone on unpaired real‑world background videos. Random masks of varied shapes, temporal dynamics, and sparsity are applied online, forcing the model to learn a pure background‑completion prior without ever seeing foreground objects or their shadows. This stage prevents the network from learning to hallucinate foreground content inside masked regions.
   Stage II fine‑tunes the model on paired synthetic data that contain objects together with their shadows and reflections. During this stage, three complementary mechanisms are applied: (i) Mask Degradation – random frame dropout (20‑99 %), morphological erosion/dilation, and coarse bounding‑box replacements simulate real‑world mask defects; (ii) DA‑Seg supervision – the side‑branch is trained to predict the degraded mask, reinforcing internal localization; (iii) MUSE – applied during temporal mask down‑sampling to correct the structural mis‑alignment introduced by the compression. The curriculum reduces optimization difficulty: Stage I supplies a strong background prior, while Stage II focuses on side‑effect suppression and robustness to weak masks.

Experimental Validation.
The authors evaluate SVOR on several benchmarks: RORD‑50 (a newly introduced real‑world paired test set), DAVIS, and the ROSE‑Bench, together with degraded‑mask variants of each. Baselines include MiniMax‑Remover, ROSE, and other diffusion‑based inpainting models. SVOR consistently outperforms baselines in PSNR/SSIM (average gains of ~1.2 dB / 0.03), Temporal Warping Error (‑35 %), and Flicker Score (‑40 %). Ablation studies confirm that removing MUSE leads to a 30 % increase in under‑erasure for fast‑motion clips, while disabling DA‑Seg degrades mask‑localization accuracy under high noise by ~15 %. The two‑stage curriculum yields a 10 % boost in shadow‑removal quality compared to a single‑stage training regime.

Strengths.

• Problem‑driven design: The paper clearly identifies three concrete failure modes and proposes targeted solutions rather than a monolithic architecture.
• Simplicity and efficiency: MUSE is a parameter‑free mask‑union operation; DA‑Seg adds only a lightweight side‑branch; both integrate seamlessly into existing DiT‑based pipelines.
• Robust training strategy: Leveraging abundant unpaired background videos for Stage I dramatically reduces reliance on costly paired data, while synthetic paired data with mask degradation bridges the domain gap.
• Comprehensive evaluation: Multiple datasets, degraded‑mask benchmarks, and thorough ablations substantiate the claims.

Weaknesses / Open Questions.

• Fixed temporal window: MUSE uses a static window size; extremely long motions or variable frame rates may still suffer from mask loss. Adaptive windowing could further improve robustness.
• Side‑branch overhead: Although lightweight, DA‑Seg still introduces extra parameters and memory consumption, which may hinder real‑time deployment on edge devices.
• Synthetic bias in Stage II: The fine‑tuning stage still depends on synthetic object‑shadow pairs; residual domain shift may appear in highly complex real scenes with indirect lighting.
• User interaction: The framework assumes masks are supplied externally. Integrating an interactive mask refinement loop could make the system more practical for end‑users.

Conclusion and Future Directions.
SVOR represents a significant step toward practical video object removal by jointly addressing mask loss, mask degradation, and side‑effect suppression. Its modular components (MUSE, DA‑Seg, curriculum training) can be incorporated into other diffusion‑based video editing systems. Future work may explore dynamic window sizing for MUSE, ultra‑lightweight segmentation heads, and closed‑loop user‑guided mask correction to push SVOR toward real‑time, consumer‑grade video editing tools.