Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction

The paper introduces **ChartIR**, a new method designed to improve how Multimodal Large Language Models (MLLMs) generate executable code from visual charts. While MLLMs are good at visual tasks, they often struggle with chart-to-code generation because it requires both precise visual understanding and accurate translation of visual elements into structured code. ChartIR addresses this by breaking down the complex process into two main stages: **initial code generation** and **iterative refinement**. In the first stage, it uses a **structured description** of the reference chart to help the MLLM understand the visual elements, such as plot type, axes, colors, and annotations, before generating an initial version of the code. Then, in the iterative refinement stage, ChartIR repeatedly compares the chart generated from the current code with the original reference chart, identifies **specific visual differences** (like color or text errors), and uses this information to progressively improve the code. Unlike other methods that might focus on only one evaluation metric, ChartIR considers all visual and structural aspects holistically during refinement, updating the code only when it quantifiably reduces the discrepancy across multiple metrics. This **training-free and model-agnostic** approach has shown **superior performance** compared to direct code generation and other state-of-the-art methods like METAL across various benchmarks and with both open-source models (like Qwen2-VL) and powerful closed-source models (like GPT-4o), demonstrating significant gains in the visual and structural accuracy of the generated charts.

https://arxiv.org/pdf/2506.14837

Видео Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction канала AI Papers Podcast Daily