A Survey of AI-Generated Video Evaluation

The growing capabilities of AI in generating video content have brought forward significant challenges in effectively evaluating these videos. Unlike static images or text, video content involves complex spatial and temporal dynamics which may require a more comprehensive and systematic evaluation of its contents in aspects like video presentation quality, semantic information delivery, alignment with human intentions, and the virtual-reality consistency with our physical world. This survey identifies the emerging field of AI-Generated Video Evaluation (AIGVE), highlighting the importance of assessing how well AI-generated videos align with human perception and meet specific instructions. We provide a structured analysis of existing methodologies that could be potentially used to evaluate AI-generated videos. By outlining the strengths and gaps in current approaches, we advocate for the development of more robust and nuanced evaluation frameworks that can handle the complexities of video content, which include not only the conventional metric-based evaluations, but also the current human-involved evaluations, and the future model-centered evaluations. This survey aims to establish a foundational knowledge base for both researchers from academia and practitioners from the industry, facilitating the future advancement of evaluation methods for AI-generated video content.

💡 Research Summary

The paper introduces and formalizes the emerging research area of AI‑Generated Video Evaluation (AIGVE), arguing that evaluating AI‑generated video requires more than traditional Video Quality Assessment (VQA) because videos combine spatial complexity with temporal dynamics and must also satisfy textual instructions. After motivating the need—citing rapid advances in models such as ChatGPT, Sora, LLaMA, and Meta Movie Gen—the authors enumerate six prevalent error categories observed in generated videos: technical (low resolution, blur, artifacts), dynamic (lack of meaningful motion), physical (violations of real‑world physics), consistency (unexpected changes in object identity or appearance), quality (structural distortions), and alignment (deviation from the prompt).

AIGVE is structured around two complementary criteria: (1) alignment with human perception, which covers visual fidelity, temporal coherence, physical realism, and overall perceptual believability; and (2) alignment with human instructions, which assesses how faithfully the video follows the supplied textual prompt. The survey maps existing evaluation tools onto these criteria. Conventional VQA metrics (PSNR, SSIM, VMAF) address low‑level technical errors, while newer multimodal metrics such as CLIPScore, GPT‑4V, ImageReward, and other vision‑language models target cross‑modal alignment. The authors further distinguish metric‑based evaluation (re‑using or designing new quantitative scores) from model‑based evaluation (training neural evaluators on human‑rated data to mimic human judgment).

The paper reviews current benchmark datasets, emphasizing the shift from simple video‑opinion pairs to richer video‑instruction‑opinion triplets. It categorizes benchmarks into five groups based on focus (overall perceptual quality, temporal dynamics, semantic reasoning, real‑world safety, etc.), and details the standard data‑collection pipeline: instruction gathering, video generation by multiple text‑to‑video models, and human scoring (0–5).

Sections 4 and 5 provide in‑depth analyses of the two evaluation dimensions, listing representative datasets, metrics, and model‑based approaches for each. The authors note that most existing metrics capture only a subset of the six error types, and that instruction‑alignment remains under‑explored.

Finally, the survey outlines open challenges and future directions: (i) unified multimodal frameworks that jointly model visual, linguistic, and temporal information; (ii) interpretable evaluation scores that explain failure modes; (iii) ethical and safety considerations such as bias, misinformation, and harmful content; and (iv) scalable, real‑time evaluation pipelines. By consolidating fragmented literature and proposing a structured roadmap, the paper aims to serve as a foundational reference for both academic researchers and industry practitioners seeking robust, nuanced methods to assess AI‑generated video content.