Inference-time Stochastic Refinement of GRU-Normalizing Flow for Real-time Video Motion Transfer

Real-time video motion transfer applications such as immersive gaming and vision-based anomaly detection require accurate yet diverse future predictions to support realistic synthesis and robust downstream decision making under uncertainty. To improve the diversity of such sequential forecasts we propose a novel inference-time refinement technique that combines Gated Recurrent Unit-Normalizing Flows (GRU-NF) with stochastic sampling methods. While GRU-NF can capture multimodal distributions through its integration of normalizing flows within a temporal forecasting framework, its deterministic transformation structure can limit expressivity. To address this, inspired by Stochastic Normalizing Flows (SNF), we introduce Markov Chain Monte Carlo (MCMC) steps during GRU-NF inference, enabling the model to explore a richer output space and better approximate the true data distribution without retraining. We validate our approach in a keypoint-based video motion transfer pipeline, where capturing temporally coherent and perceptually diverse future trajectories is essential for realistic samples and low bandwidth communication. Experiments show that our inference framework, Gated Recurrent Unit- Stochastic Normalizing Flows (GRU-SNF) outperforms GRU-NF in generating diverse outputs without sacrificing accuracy, even under longer prediction horizons. By injecting stochasticity during inference, our approach captures multimodal behavior more effectively. These results highlight the potential of integrating stochastic dynamics with flow-based sequence models for generative time series forecasting. The code is available at: https://github.com/Tasmiah1408028/Inference-Time-Stochastic-Refinement-Of-GRU-NF-For-Real-Time-Video-Motion-Transfer

💡 Research Summary

The paper addresses the need for diverse yet accurate future predictions in real‑time video motion‑transfer systems, such as immersive VR gaming and vision‑based anomaly detection. Existing approaches either suffer from mode collapse (e.g., variational recurrent networks) or from the topological constraints of normalizing‑flow models, which limit their ability to capture well‑separated multimodal futures, especially over long horizons.

To overcome these limitations, the authors propose GRU‑Stochastic Normalizing Flow (GRU‑SNF), an inference‑time refinement of a pre‑trained GRU‑Normalizing Flow (GRU‑NF) model. The core idea is to insert a small number of Markov‑Chain Monte‑Carlo (MCMC) steps between the layers of the normalizing flow during generation. Each MCMC step proposes a perturbation from a simple Gaussian proposal distribution and accepts or rejects it using a Metropolis‑Hastings criterion based on a composite energy function

\