Ontology-Guided Neuro-Symbolic Inference: Grounding Language Models with Mathematical Domain Knowledge

Language models exhibit fundamental limitations – hallucination, brittleness, and lack of formal grounding – that are particularly problematic in high-stakes specialist fields requiring verifiable reasoning. I investigate whether formal domain ontologies can enhance language model reliability through retrieval-augmented generation. Using mathematics as proof of concept, I implement a neuro-symbolic pipeline leveraging the OpenMath ontology with hybrid retrieval and cross-encoder reranking to inject relevant definitions into model prompts. Evaluation on the MATH benchmark with three open-source models reveals that ontology-guided context improves performance when retrieval quality is high, but irrelevant context actively degrades it – highlighting both the promise and challenges of neuro-symbolic approaches.

💡 Research Summary

This paper investigates whether formal domain ontologies can improve the reliability of small to medium-sized language models (≤ 9 B parameters) by grounding their reasoning in mathematically verified knowledge. The authors focus on mathematics as a clean testbed and adopt the OpenMath standard, which provides machine‑readable Content Dictionaries (CDs) containing symbol definitions, natural‑language descriptions, and Formal Mathematical Properties (FMPs). They construct a neuro‑symbolic pipeline that consists of five stages: (1) building an OpenMath knowledge base, (2) extracting key mathematical concepts from a natural‑language problem, (3) performing hybrid retrieval (BM25 + dense embeddings) to fetch candidate definitions, (4) re‑ranking the candidates with a cross‑encoder, and (5) augmenting the model prompt with the top‑ranked definitions.

The central hypothesis is that augmenting a problem P with relevant definitions R(P, K) yields a new inference function  â = M(P, R(P, K)), which should increase accuracy when retrieval quality is high and cause degradation when irrelevant symbols are injected. To test this, the authors evaluate three open‑source models: Gemma‑2B (2.6 B), Gemma‑9B (9.2 B), and Qwen2.5‑Math‑7B (7.6 B, fine‑tuned on mathematics). They use the MATH‑500 subset (500 problems spanning seven types and five difficulty levels) and compare two conditions: a baseline (problem + instructions only) and an OpenMath‑augmented condition.

Two inference modes are examined: Greedy (single deterministic generation) and Best‑of‑N (up to five samples at temperature 0.6). Primary metrics are answer accuracy and average number of attempts required to obtain a correct answer. The authors also vary the cross‑encoder relevance threshold from 0.0 to 0.9 to study how filtering noise affects performance.

Results show a nuanced picture. The math‑specialized Qwen2.5‑Math‑7B consistently benefits from OpenMath definitions across all thresholds, with especially large gains in Algebra (+2.4 % to +13.3 %) and Geometry (+33.3 % at the highest threshold). Gemma‑9B experiences a transition: low thresholds (many noisy definitions) cause slight accuracy drops, but higher thresholds (stricter relevance) lead to positive deltas, indicating that the model can exploit high‑quality definitions once noise is removed. Gemma‑2B, however, fails to leverage the external knowledge; its accuracy declines monotonically as the threshold increases, suggesting insufficient capacity to parse and integrate formal definitions.

Difficulty‑level analysis reveals that mid‑range problems (levels 2‑4) receive the greatest benefit, while the easiest level (1) often sees degradation because the models already know the required concepts. At the hardest level (5), Qwen2.5‑Math‑7B’s internal expertise sometimes conflicts with external definitions, leading to a “specialization paradox” where added context harms performance, whereas Gemma‑9B can improve in Best‑of‑N mode.

Problem‑type breakdown shows strong positive effects for Algebra and Geometry, moderate or mixed effects for Number Theory (high-quality definitions concentrated in easy problems), and variable patterns for Pre‑Calculus.

Efficiency analysis indicates that OpenMath generally reduces the average number of attempts in Best‑of‑N mode, especially for mid‑difficulty problems, implying that formal definitions provide reasoning shortcuts. However, at the highest difficulty level, attempts decrease while accuracy may still drop, hinting at a “false confidence” effect where models converge faster to incorrect answers.

Overall, the study confirms that ontology‑guided retrieval can enhance the reasoning reliability of resource‑constrained language models, but only when retrieval quality is high, the model has enough capacity to process the added semantics, and the problem domain aligns with the ontology’s coverage. The authors suggest future work on tighter integration with symbolic engines (e.g., SymPy), dynamic context selection, and extending the approach to other high‑stakes domains such as medicine or law.