Autonomous Data Processing using Meta-Agents

Traditional data processing pipelines are typically static and handcrafted for specific tasks, limiting their adaptability to evolving requirements. While general-purpose agents and coding assistants can generate code for well-understood data pipelines, they lack the ability to autonomously monitor, manage, and optimize an end-to-end pipeline once deployed. We present \textbf{Autonomous Data Processing using Meta-agents} (ADP-MA), a framework that dynamically constructs, executes, and iteratively refines data processing pipelines through hierarchical agent orchestration. At its core, \textit{meta-agents} analyze input data and task specifications to design a multi-phase plan, instantiate specialized \textit{ground-level agents}, and continuously evaluate pipeline performance. The architecture comprises three key components: a planning module for strategy generation, an orchestration layer for agent coordination and tool integration, and a monitoring loop for iterative evaluation and backtracking. Unlike conventional approaches, ADP-MA emphasizes context-aware optimization, adaptive workload partitioning, and progressive sampling for scalability. Additionally, the framework leverages a diverse set of external tools and can reuse previously designed agents, reducing redundancy and accelerating pipeline construction. We demonstrate ADP-MA through an interactive demo that showcases pipeline construction, execution monitoring, and adaptive refinement across representative data processing tasks.

💡 Research Summary

The paper introduces ADP‑MA (Autonomous Data Processing using Meta‑Agents), a framework that automatically constructs, executes, and iteratively refines multi‑stage data processing pipelines from natural‑language task specifications and raw tabular inputs. The authors argue that existing workflow engines (Airflow, Prefect) and large‑language‑model (LLM) coding assistants each address only a fragment of the problem: workflow engines require manually defined DAGs, while LLM assistants generate isolated code snippets without handling cascading failures, data‑quality validation, or cost‑aware exploration.

ADP‑MA’s core innovation is a hierarchical orchestration architecture composed of three persistent meta‑agents (Orchestrator, Architect, Monitor) and a pool of transient ground‑level agents (Reader, Transformer, Partitioner, Indexer, Code Generator). The Orchestrator receives the user’s natural‑language goal and input datasets, profiles the data (schema inference, column statistics, distribution summaries), and decomposes the goal into a logical sequence of phases (e.g., profiling → cleaning → join → feature engineering → model training). The Architect translates this logical plan into a physical execution plan by selecting concrete agent types, code generation strategies (e.g., Pandas vs. Polars), and estimating resource costs.

Ground agents are instantiated on demand: each receives a tailored prompt, calls an LLM to generate Python/DataFrame code, and runs the code inside a sandboxed process with namespace isolation. Before committing to full‑scale execution, ADP‑MA employs progressive sampling (testing on 1 %, 5 %, 25 % of the data) to catch errors early and control monetary latency costs associated with LLM calls.

Reliability is ensured through four complementary mechanisms:

1. Progressive Sampling – cheap validation on increasingly larger data subsets, acting as a cost‑based optimizer.
2. Schema Contracts – dynamically generated per‑stage contracts that declare expected column names, types, and invariants; these are automatically checked at runtime, analogous to integrity constraints in databases.
3. Two‑Level Backtracking – a local backtrack that regenerates code for a failing ground agent, and a global backtrack that revises the overall plan when the failure stems from an incorrect decomposition.
4. Rule‑Based Monitoring – a lightweight, LLM‑free monitor that watches execution logs for data‑quality anomalies such as sudden row count spikes, null‑rate surges, or silent row drops, and triggers backtracking or user alerts.

The authors evaluate ADP‑MA on four diverse benchmark suites covering data‑science workflows, scientific pipelines, complex code generation, and database query tasks, totaling 556 individual tasks. Experiments span five LLM back‑ends (Claude, GPT‑4, Gemini, DeepSeek, Mistral). Across all settings, ADP‑MA outperforms single‑agent baselines (e.g., AutoKaggle, smolagents, DS‑GURU) and matches or exceeds multi‑agent systems that rely on longer compute budgets. Ablation studies reveal that removing any of the four reliability mechanisms degrades success rates by 12–25 percentage points; eliminating the two‑level backtrack alone drops overall pipeline success below 40 %. A variance analysis shows that 78 % of tasks exhibit deterministic outcomes within ±4 pp across repeated runs, indicating strong stability despite the stochastic nature of LLM generation.

Implementation details include a Google ADK‑based infrastructure, plug‑in domain knowledge packs that inject domain‑specific keywords into prompts, and a structured workspace that records data bins, metadata bins, case logs, and sampling states for reproducibility. The system’s design principles—hierarchical planning, iterative refinement, progressive sampling, intelligent backtracking, and domain‑agnostic meta‑agents—mirror classic database concepts (logical/physical planning, cost‑based optimization, integrity constraints, runtime watchdogs) but are applied to the novel setting where the query itself is a natural‑language intent.

The paper’s contributions are: (1) formal problem definition and architecture for autonomous pipeline construction; (2) a suite of system‑level reliability mechanisms that together enable robust LLM‑driven data processing; (3) comprehensive empirical validation across heterogeneous workloads and LLM families; (4) a publicly released, reproducible codebase with domain packs and evaluation scripts.

In conclusion, ADP‑MA demonstrates that a well‑engineered, systems‑oriented orchestration layer can unlock the full potential of LLMs for end‑to‑end data pipelines, achieving higher reliability and cost‑efficiency than naïve “flat” agent loops. Future work is suggested in extending the framework to distributed execution, streaming data, mixed tabular‑document pipelines, and automated semantic correctness verification without ground‑truth labels.