Herb.jl: A Unifying Program Synthesis Library

Program synthesis – the automatic generation of code given a specification – is one of the most fundamental tasks in artificial intelligence (AI) and the dream of many programmers. Numerous synthesizers have been developed for program synthesis, offering different approaches to the exponentially growing program space. Although such state-of-the-art tools exist, reusing and adapting them remains tedious and time-consuming. We propose Herb.jl, a unifying program synthesis library written in Julia, to address these issues. Since current methods share similar building blocks, we aim to break down the underlying algorithms into extendable, reusable subcomponents. To demonstrate the benefits of using Herb.jl, we show how to implement a simple problem and grammar, and how to solve it with just a few lines of code.

💡 Research Summary

The paper introduces Herb.jl, a Julia‑based, modular library that aims to unify and simplify the development, reuse, and benchmarking of program synthesis tools. Program synthesis—automatically generating code from a specification—is a core AI problem, but existing synthesizers are typically tightly coupled to specific domains, implementations, or benchmark suites, making it difficult to adapt, extend, or fairly compare them. The authors identify four recurring obstacles: (1) domain‑specific implementations that cannot be easily repurposed, (2) common building blocks (grammars, constraints, search strategies, interpreters) that are not exposed as reusable components, (3) lack of transparency in engineering choices that confound algorithmic comparisons, and (4) benchmarks that are hard to reuse because specifications and grammars are intertwined or hidden.

To address these issues, Herb.jl decomposes a synthesis problem into five abstract components: Specification, Grammar, Constraints, Interpreter, and Search. Each component is defined by an abstract type and a small set of required methods, allowing developers to plug in default implementations or provide custom ones with minimal boilerplate. The library is split into several sub‑packages: HerbCore.jl (core types and interfaces), HerbGrammar.jl (declarative CFG definition via the @csgrammar macro), HerbConstraints.jl (semantic constraints beyond context‑free grammars), HerbInterpret.jl (AST execution and interaction with external simulators), and HerbSearch.jl (top‑down, bottom‑up, stochastic, and hybrid search strategies).

A key design decision is that grammars are first‑class objects that can be extended at runtime, enabling rapid prototyping of new domain‑specific languages. Constraints are expressed as predicate functions over partial programs, allowing pruning of semantically irrelevant candidates without altering the underlying CFG. The interpreter abstracts program execution, so users can attach a simple evaluator for toy languages or a full‑fledged hardware simulator for low‑level code generation. Search strategies are implemented as composable pipelines; for example, a CEGIS loop can be built by combining a counterexample generator, a constraint solver, and a top‑down enumerator, all supplied by the library.

To demonstrate practicality, the authors present three use cases. First, they show how to encode a simple integer‑arithmetic problem (with examples (0→1, 1→3, …)) using HerbSpecification.jl, define a grammar with literals, variable x, addition, and multiplication, and run a basic enumerative search—all in under ten lines of Julia code. Second, they illustrate how an existing synthesizer can be re‑implemented in Garden.jl, a companion repository that contains reference implementations of well‑known techniques (e.g., CEGIS, Sketch, neural‑symbolic approaches) built on top of Herb.jl’s abstractions. By swapping out a heuristic cost function or replacing a constraint solver, researchers can experiment with new ideas without rewriting the whole system. Third, they load a suite of human‑readable benchmark problems from HerbBenchmarks.jl, which stores each benchmark as a self‑contained specification‑grammar pair, and run multiple synthesizers from Garden.jl against the same benchmark set, enabling fair, reproducible performance comparisons.

The ecosystem is deliberately open: all sub‑packages are published on the Julia package registry, and their source code is hosted on GitHub. The meta‑package Herb.jl aggregates the core modules for end‑users, while Garden.jl and HerbBenchmarks.jl are optional add‑ons that can be loaded as needed. This modular distribution encourages both novice users—who can start with the high‑level API to solve small synthesis tasks—and advanced researchers—who can dive into the low‑level interfaces to develop novel algorithms or integrate external solvers.

In summary, Herb.jl offers a principled, extensible framework that abstracts the common ingredients of program synthesis into reusable modules, provides reference implementations for rapid prototyping, and supplies a standardized benchmark suite for reproducible evaluation. By leveraging Julia’s performance and its rich scientific‑computing ecosystem, Herb.jl bridges the gap between theoretical synthesis research and practical, real‑world applications, fostering more transparent, comparable, and reusable advances in the field.