Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation

In last two years, large language models (LLMs) have shown strong capabilities in code generation, including hardware design at register-transfer level (RTL). While their use in high-level synthesis (HLS) remains comparatively less mature, the ratio of HLS- to RTL-focused studies has shifted from 1:10 to 2:10 in the past six months, indicating growing interest in leveraging LLMs for high-level design entry while relying on downstream synthesis for optimization. This growing trend highlights the need for a comprehensive benchmarking and evaluation framework dedicated to LLM-based HLS. To address this, We present Bench4HLS for evaluating LLM-generated HLS designs. Bench4HLS comprises 170 manually drafted and validated case studies, spanning small kernels to complex accelerators, curated from widely used public repositories. The framework supports fully automated assessment of compilation success, functional correctness via simulation, and synthesis feasibility/optimization. Crucially, Bench4HLS integrates a pluggable API for power, performance, and area (PPA) analysis across various HLS toolchains and architectures, demonstrated here with Xilinx Vitis HLS and validated on Catapult HLS. By providing a structured, extensible, and plug-and-play testbed, Bench4HLS establishes a foundational methodology for benchmarking LLMs in HLS workflows.

💡 Research Summary

The paper introduces Bench4HLS, a comprehensive benchmarking framework designed to evaluate large language model (LLM) generated high‑level synthesis (HLS) code. Recognizing that existing benchmarks such as HLS‑Eval focus on small kernels, lack power‑performance‑area (PPA) analysis, and support only a limited number of tools, the authors construct a much larger and more diverse dataset of 170 manually curated HLS designs. Each entry consists of a natural‑language instruction, a synthesizable C/C++ HLS implementation, and a corresponding testbench, with an average of 88 lines of code per case, covering everything from simple kernels to full accelerator‑scale applications sourced from repositories like CHStone, HLS4ML, and Rosetta.

Bench4HLS provides a fully automated evaluation pipeline that proceeds through four stages: (1) compilation to catch syntax errors, (2) pre‑synthesis C‑simulation using the supplied testbench for functional correctness, (3) synthesis with Xilinx Vitis HLS (and optional Catapult HLS) to generate RTL, and (4) post‑synthesis simulation and RTL‑level validation. The pipeline is designed to be pluggable, allowing other HLS toolchains and downstream place‑and‑route tools (e.g., Vivado) to be integrated without code changes.

A key contribution is the modular PPA API. After synthesis, the framework automatically extracts latency, clock period, register, DSP, BRAM, and power metrics from the target toolchain, enabling side‑by‑side comparison of QoR across different tools and design variants. In addition, Bench4HLS embeds a design‑space exploration (DSE) engine that systematically varies pragmas, pipeline depths, and memory partitioning under user‑defined resource or timing budgets, reporting Pareto‑optimal trade‑offs such as ΔLatency, ΔFF utilization, and ΔPower.

The authors also provide a standardized Python API for integrating LLMs. They demonstrate the framework with several models—including GPT‑5, QwenCoder, and Llama—showing how each model can be queried with the natural‑language instruction, how the generated code is post‑processed, and how the resulting designs are fed through the full evaluation flow. The framework records compilation failures, functional mismatches, synthesis failures, and PPA degradations, offering a holistic view of each model’s strengths and weaknesses.

Compared to prior work, Bench4HLS expands the benchmark size (170 vs. ≤94 cases), adds full PPA analysis, supports multiple commercial HLS tools, and incorporates automated DSE and multi‑stage functional verification. This makes it the first end‑to‑end, practically meaningful benchmark for LLM‑driven HLS workflows. The authors argue that such a framework is essential for advancing LLM‑assisted hardware design, as it provides the necessary data, tooling, and metrics to guide future research toward fully automated, high‑quality FPGA accelerator generation without manual expert intervention.