A Versatile Multimodal Agent for Multimedia Content Generation

With the advancement of AIGC (AI-generated content) technologies, an increasing number of generative models are revolutionizing fields such as video editing, music generation, and even film production. However, due to the limitations of current AIGC models, most models can only serve as individual components within specific application scenarios and are not capable of completing tasks end-to-end in real-world applications. In real-world applications, editing experts often work with a wide variety of images and video inputs, producing multimodal outputs – a video typically includes audio, text, and other elements. This level of integration across multiple modalities is something current models are unable to achieve effectively. However, the rise of agent-based systems has made it possible to use AI tools to tackle complex content generation tasks. To deal with the complex scenarios, in this paper, we propose a MultiMedia-Agent designed to automate complex content creation. Our agent system includes a data generation pipeline, a tool library for content creation, and a set of metrics for evaluating preference alignment. Notably, we introduce the skill acquisition theory to model the training data curation and agent training. We designed a two-stage correlation strategy for plan optimization, including self-correlation and model preference correlation. Additionally, we utilized the generated plans to train the MultiMedia-Agent via a three stage approach including base/success plan finetune and preference optimization. The comparison results demonstrate that the our approaches are effective and the MultiMedia-Agent can generate better multimedia content compared to novel models.

💡 Research Summary

The paper presents MultiMedia‑Agent, a novel multimodal agent designed to automate the creation of complex multimedia content that integrates images, video, audio, speech, and text. Recognizing that existing AI‑generated content (AIGC) models are typically isolated components that cannot handle end‑to‑end workflows, the authors draw inspiration from human Skill Acquisition Theory (cognitive, associative, autonomous stages) to construct a three‑stage training pipeline that progressively equips the agent with increasingly sophisticated capabilities.

Data Generation Pipeline
The authors first build a “multimedia content playground” consisting of 18 realistic tasks (e.g., converting a series of photos into a wedding slideshow, merging photos with background music to produce a travel memory video). Publicly available multimedia data are collected, and GPT‑4o is employed to synthesize diverse user queries that pair each task with appropriate input media. This yields a rich dataset of multimodal requests, each linked to a concrete set of media files.

Tool Library
A comprehensive tool library is assembled, organized into three categories: (1) Multimodal Understanding tools (five modality‑specific “any‑to‑text” models), (2) Generative/Editing tools (image, video, audio, speech generation and editing, plus non‑deep‑learning utilities such as transition effects), and (3) Auxiliary tools (video concatenation, audio‑video synchronization, retrieval, etc.). Each tool is described in a JSON schema that includes the tool name, model name, fixed input/output file extensions (.png, .mp4, .mp3, .txt), required parameters with type and description, and a textual description of functionality. This explicit schema reduces format errors during plan generation.

Hierarchical Plan Curation
Plans are generated by prompting GPT‑4o with the user query, available media, and the tool library. A plan is a list of dictionaries, each specifying a tool call and its parameters. To ensure both execution feasibility and aesthetic quality, the authors introduce a two‑stage correlation strategy:

1. Self‑Correlation – GPT‑4o reviews its own base plan, identifies logical or technical flaws, and produces a self‑corrected version.
2. Preference Correlation – The self‑corrected plan is executed, producing multimedia output. A set of preference‑based evaluation models (trained on human judgments) scores the output on criteria such as visual‑audio coherence, narrative flow, and user‑aligned aesthetics. Based on these scores, GPT‑4o iteratively refines the plan to maximize preference alignment.

Three‑Stage Training (Agent Skill Acquisition)

• Cognitive Stage: The agent is fine‑tuned on all generated plans (base, self‑corrected, preference‑optimized). This stage teaches the agent basic tool semantics, input/output handling, and simple sequencing, analogous to a beginner learning fundamentals.
• Associative Stage: Fine‑tuning is restricted to successful plans only. The agent learns higher‑level workflow composition, error handling, and inter‑tool dependencies, mirroring targeted practice.
• Autonomous Stage: Preference data derived from the model’s own evaluations are used for reinforcement‑style fine‑tuning. The agent internalizes human aesthetic preferences, enabling it to generate content that not only works technically but also satisfies subjective quality criteria.

Experiments and Comparisons
The MultiMedia‑Agent is benchmarked against several notable tool‑based agents: HuggingGPT, ToolLLM, NExT‑GPT, ModaVerse, and AutoDirector. Evaluation metrics include (i) tool execution success rate, (ii) multimodal content quality (visual‑audio‑text consistency, narrative coherence), and (iii) preference alignment scores. Across the 18 tasks, MultiMedia‑Agent consistently achieves higher success rates and superior preference scores, especially in scenarios requiring tight coupling of multiple modalities (e.g., image+audio→video with subtitles).

Contributions

1. A full pipeline for generating multimodal content data, comprising a realistic task suite, a rich tool library, and preference‑driven evaluation metrics.
2. A two‑stage plan curation method that combines self‑reflection and human‑preference optimization to produce high‑quality execution plans.
3. An agent training framework grounded in Skill Acquisition Theory, enabling progressive learning from basic tool usage to autonomous, preference‑aware content creation.

Significance
The work demonstrates that multimodal AIGC systems can move beyond isolated generators to become autonomous agents capable of end‑to‑end multimedia production. By mirroring human learning stages and incorporating explicit human‑preference feedback, the proposed architecture bridges the gap between functional correctness and user‑centric quality. The publicly described tool library and plan‑generation methodology provide a reusable foundation for future research in multimodal agents, while the preference‑alignment component offers a concrete path toward more human‑aligned generative AI.