Explaining the Explainer: Understanding the Inner Workings of Transformer-based Symbolic Regression Models

Following their success across many domains, transformers have also proven effective for symbolic regression (SR); however, the internal mechanisms underlying their generation of mathematical operators remain largely unexplored. Although mechanistic interpretability has successfully identified circuits in language and vision models, it has not yet been applied to SR. In this article, we introduce PATCHES, an evolutionary circuit discovery algorithm that identifies compact and correct circuits for SR. Using PATCHES, we isolate 28 circuits, providing the first circuit-level characterisation of an SR transformer. We validate these findings through a robust causal evaluation framework based on key notions such as faithfulness, completeness, and minimality. Our analysis shows that mean patching with performance-based evaluation most reliably isolates functionally correct circuits. In contrast, we demonstrate that direct logit attribution and probing classifiers primarily capture correlational features rather than causal ones, limiting their utility for circuit discovery. Overall, these results establish SR as a high-potential application domain for mechanistic interpretability and propose a principled methodology for circuit discovery.

💡 Research Summary

The paper tackles the largely unexplored problem of mechanistic interpretability for transformer‑based symbolic regression (SR) models. While transformers have become state‑of‑the‑art for generating symbolic expressions, the internal circuitry that decides which mathematical operators (e.g., sin, +, ×) to emit has never been systematically dissected. To fill this gap, the authors introduce PATCHES (Probabilistic Algorithm for Tuning Circuits through Heuristic Evolution and Search), an evolutionary circuit‑discovery framework that searches for sparse sub‑graphs (circuits) of the model that are both minimal and functionally sufficient.

PATCHES treats every attention head and MLP block as a binary inclusion variable. Using the Covariance Matrix Adaptation Evolution Strategy (CMA‑ES), it samples probabilistic masks, evaluates each candidate by a dual‑objective fitness function that penalises circuit size and performance loss, and iteratively refines the distribution. The performance loss is measured against a set of token‑specific datasets (e.g., all equations containing the token “sin” but not “cos”) and is quantified either by functional metrics (top‑k accuracy) or model‑level metrics (logit differences). The fitness function is F(C)=|C|+λ · ∑_i max(0, T_i−S_i(C)), where λ=100 and T_i are predefined performance thresholds.

A central methodological contribution is the systematic comparison of three patching strategies used to create counterfactual activations during evaluation: (1) Mean patching, which replaces a component’s activation with its dataset‑wide mean; (2) Resample patching, which constructs corrupted inputs by swapping the target token with all other possible unary or binary tokens and averages the resulting activations; and (3) Resample Symmetric Token Replacement (STR), a single‑sample variant that uses the activation of the most semantically similar token. Empirical results show that mean patching yields the most stable and reliable causal signals, while resample methods are more sensitive to the choice of corrupted tokens.

To verify that discovered circuits truly explain the model’s behavior, the authors formalise three evaluation criteria: Faithfulness (the circuit alone reproduces the full model’s predictions), Completeness (removing the circuit causes a significant drop in performance), and Minimality (each component of the circuit is indispensable). These criteria are applied both at the functional level (top‑k accuracy) and the model level (logit score differences), providing a rigorous, multi‑faceted validation pipeline.

Applying PATCHES to the NeSymReS transformer (a canonical encoder‑decoder SR architecture) the authors discover 28 distinct circuits, each responsible for generating a specific unary or binary operator. The circuits are remarkably compact—often only a handful of heads and MLPs—yet they achieve near‑perfect fidelity to the full model. The paper also benchmarks two popular interpretability tools: Direct Logit Attribution and probing classifiers. Both methods correlate strongly with the presence of relevant information but fail to capture the causal influence demonstrated by the PATCHES‑derived circuits; removing components identified by probing does not substantially degrade performance, underscoring their primarily correlational nature.

In summary, the contributions are: (1) the first mechanistic analysis of transformer‑based symbolic regression, revealing concrete sub‑networks that implement mathematical operators; (2) the PATCHES algorithm, which leverages CMA‑ES to efficiently discover minimal, high‑fidelity circuits; (3) a principled evaluation framework based on faithfulness, completeness, and minimality, validated across functional and model‑level metrics; (4) an empirical comparison showing mean patching as the most reliable causal intervention; and (5) a critical assessment demonstrating the limitations of logit‑based attribution and probing for causal circuit discovery.

The work opens a new avenue for interpreting scientific AI systems, suggesting that evolutionary circuit search combined with rigorous causal testing can render even complex, black‑box models transparent. Future research may extend PATCHES to larger multimodal models, explore automated generation of token‑specific datasets, and integrate the discovered circuits into model compression or safety‑critical applications.