ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoning

With their high information density and intuitive readability, charts have become the de facto medium for data analysis and communication across disciplines. Recent multimodal large language models (MLLMs) have made notable progress in automated chart understanding, yet they remain heavily dependent on explicit textual annotations and the performance degrades markedly when key numerals are absent. To address this limitation, we introduce ChartAgent, a chart understanding framework grounded in Tool-Integrated Reasoning (TIR). Inspired by human cognition, ChartAgent decomposes complex chart analysis into a sequence of observable, replayable steps. Supporting this architecture is an extensible, modular tool library comprising more than a dozen core tools, such as keyelement detection, instance segmentation, and optical character recognition (OCR), which the agent dynamically orchestrates to achieve systematic visual parsing across diverse chart types. Leveraging TIRs transparency and verifiability, ChartAgent moves beyond the black box paradigm by standardizing and consolidating intermediate outputs into a structured Evidence Package, providing traceable and reproducible support for final conclusions. Experiments show that ChartAgent substantially improves robustness under sparse annotation settings, offering a practical path toward trustworthy and extensible systems for chart understanding.

💡 Research Summary

The paper “ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoning” introduces a novel framework designed to overcome the significant limitations of current Multimodal Large Language Models (MLLMs) in chart understanding. While MLLMs have advanced automated chart analysis, their performance heavily relies on explicit textual annotations within charts and degrades markedly in real-world scenarios where such annotations are sparse or absent (e.g., unannotated charts). ChartAgent addresses this core issue by moving away from the opaque, end-to-end mapping paradigm of typical MLLMs.

Inspired by the structured, step-by-step cognitive process of human analysts, ChartAgent is grounded in Tool-Integrated Reasoning (TIR). Its core innovation is decomposing complex chart analysis into a sequence of observable, controllable, and replayable steps. To operationalize this, the authors construct an extensible, modular tool library comprising over a dozen specialized tools. These are categorized into: 1) Low-level Visual Perception Tools (e.g., Key Element Detection, Instance Segmentation, OCR, Auxiliary Line Drawing) for fundamental visual parsing and geometric measurement, and 2) High-level Reasoning & Integration Tools (e.g., Numerical Calculation, Relational Reasoning, Data Structuring) for information synthesis and analysis.

ChartAgent functions as an intelligent orchestrator. Given a chart image and a user query (e.g., “What is the percentage of ‘Once a Day’?”), it first recognizes the chart type and decomposes the task. It then dynamically selects and invokes a sequence of tools from its library based on a cost-information gain trade-off. A critical feature is the generation of a standardized Evidence Package. All intermediate outputs from every tool execution—such as segmentation masks, detected coordinates, extracted numerical values, and drawn auxiliary lines—are consolidated into this structured package. This provides traceable, verifiable, and reproducible support for the final conclusion, moving decisively beyond the “black box” model.

In the final decision phase, ChartAgent employs a GroupTalk-based multi-expert collaboration mechanism to perform consensus voting and confidence fusion over outputs from multiple tools, yielding the final answer. The entire framework operates in a training-free, zero-shot manner, leveraging the tool library and a large language model for planning and reflection.

Experiments demonstrate that ChartAgent achieves state-of-the-art performance on the ChartQA and ChartBench datasets across key tasks like NumberQA (NQA), Value Comparison, and Chart-to-Table conversion. Most importantly, it shows substantially improved robustness in sparse-annotation settings, where traditional MLLMs struggle. The work posits that this transparent, tool-based, evidence-driven approach offers a practical and trustworthy path forward for deploying chart understanding systems in precision-sensitive domains such as scientific research and finance.