Can Test-time Computation Mitigate Reproduction Bias in Neural Symbolic Regression?

Mathematical expressions play a central role in scientific discovery. Symbolic regression aims to automatically discover such expressions from given numerical data. Recently, Neural symbolic regression (NSR) methods that involve Transformers pre-trained on synthetic datasets have gained attention for their fast inference, but they often perform poorly, especially with many input variables. In this study, we analyze NSR from both theoretical and empirical perspectives and show that (1) ordinary token-by-token generation is ill-suited for NSR, as Transformers cannot compositionally generate tokens while validating numerical consistency, and (2) the search space of NSR methods is greatly restricted due to reproduction bias, where the majority of generated expressions are merely copied from the training data. We further examine whether tailored test-time strategies can reduce reproduction bias and show that providing additional information at test time effectively mitigates it. These findings contribute to a deeper understanding of the limitation of NSR approaches and provide guidance for designing more robust and generalizable methods. Code is available at https://github.com/Shun-0922/Mem-Bias-NSR .

💡 Research Summary

This paper provides a comprehensive investigation of the limitations of current neural symbolic regression (NSR) approaches that rely on Transformer models pretrained on large synthetic equation corpora. The authors first identify a fundamental mismatch between the token‑by‑token autoregressive generation used in natural‑language models and the requirements of symbolic regression, where each token must respect the numerical consistency of the whole expression. By employing concepts from circuit‑complexity theory, they prove that a bounded‑precision Transformer cannot solve the “last‑token prediction” problem – the task of selecting the most appropriate leaf token given a partially generated expression and the underlying data. Assuming the widely believed separation TC⁰ ⊊ NC¹, they show that for sufficiently large expressions no Transformer of polynomial size can guarantee the correct final token, implying that Transformers cannot perform the necessary compositional arithmetic during generation.

Empirically, the study examines the NeSymReS model, a representative NSR system, and quantifies a severe “reproduction bias”: the majority (≈70 %) of generated formulas are exact copies of expressions that appeared in the training set. This bias becomes more pronounced as the number of input variables grows, effectively collapsing the search space to a tiny subset of the theoretically possible expressions. The authors argue that this bias is orthogonal to previously reported issues such as poor extrapolation to unseen input ranges; it reflects a deeper inability of the model to synthesize novel symbolic structures.

To mitigate this bias, the paper evaluates three test‑time computation strategies. (1) Large‑beam decoding simply widens the search horizon but yields only modest reductions in copying. (2) Monte‑Carlo Tree Search (MCTS) structures the exploration as a tree and can discover slightly more diverse candidates, at the cost of substantially higher inference time. (3) The newly proposed Neural Symbolic Regression guided by Verified Subtrees (NSR‑gvs) introduces a feedback loop: after each subtree is generated, the model evaluates its numerical error on the provided dataset and uses this information to steer subsequent token predictions. This approach injects external, data‑driven signals at inference time, encouraging the model to step outside the memorized training distribution.

Experimental results show that NSR‑gvs most effectively reduces reproduction bias, cutting the proportion of copied formulas to roughly 45 % and more than doubling the rate of genuinely novel expressions compared with the baseline. However, the authors also note trade‑offs: the verification step adds 3–5× overhead to inference, and in some cases the newly generated formulas, while novel, exhibit higher numerical error than the copied ones. This highlights that novelty does not automatically translate into better fit, and that balancing exploration and accuracy remains an open challenge.

The contributions of the work are threefold: (i) a formal proof that standard Transformers lack the compositional capability needed for numerically consistent symbolic generation, (ii) a thorough empirical demonstration of reproduction bias in state‑of‑the‑art NSR models, and (iii) a systematic comparison of test‑time strategies, with NSR‑gvs emerging as the most promising method for bias mitigation.

The paper concludes by outlining future research directions: designing generation mechanisms that operate on higher‑level symbolic units (e.g., subtrees or operator blocks) rather than individual tokens, integrating numerical verification into the training objective (e.g., via reinforcement learning or differentiable solvers), and extending evaluation to real scientific datasets in physics, chemistry, and climate science. Such advances could transform NSR from a fast but brittle inference tool into a robust, discovery‑oriented symbolic regression framework.