VERIFY-RL: Verifiable Recursive Decomposition for Reinforcement Learning in Mathematical Reasoning

Training language models to solve complex mathematical problems benefits from curriculum learning progressively training on simpler subproblems. However, existing decomposition methods are often heuristic, offering no guarantees that subproblems are simpler, that solving them aids the parent task, or that their relationships are mathematically grounded. We observe that symbolic differentiation provides a natural structure for verified decomposition: calculus rules explicitly define how expressions reduce to simpler components with provable properties. We introduce Verify-RL, a framework where every parent-child decomposition satisfies three verifiable conditions: strictly decreasing structural complexity, solution containment, and formal rule derivation. Unlike heuristic methods where a significant fraction of decompositions are invalid our properties admit automatic verification through symbolic computation, achieving “verification by construction” Experiments demonstrate that eliminating invalid decompositions yields sizable gains, accuracy on the hardest problems more than doubles from 32% to 68%, with a 40% relative improvement overall.

💡 Research Summary

The paper introduces Verify‑RL, a novel framework that brings formal verification to the recursive decomposition of mathematical reasoning tasks, specifically symbolic differentiation. Existing curriculum‑learning approaches for large language models (LLMs) often rely on heuristic generation of easier sub‑problems, but they provide no guarantees that the sub‑problems are truly simpler, that solving them contributes to the parent problem, or that the parent‑child relationship is mathematically sound. Verify‑RL addresses these gaps by grounding decomposition in the well‑defined calculus rules (chain, product, and sum) and by enforcing three verifiable properties for every parent‑child edge:

1. V1 – Easier Child: The structural complexity, measured as the nesting depth of the expression tree, must strictly decrease for chain and product decompositions (non‑strict for sum). This ensures a clear curriculum ordering where children are always learned before their parents.

2. V2 – Solution Helpful: The derivative (solution) of the child must appear as a factor or additive term in the derivative of the parent. In other words, solving the child provides a concrete component of the parent’s answer.

3. V3 – Rule Derivation: The child must be a syntactic subtree of the parent and the relationship must be derivable from a specific calculus rule (chain, product, or sum).

These properties are automatically checked using a symbolic engine (e.g., SymPy). If a decomposition fails any check, it is discarded and the system attempts an alternative decomposition, achieving “verification by construction.”

The authors formalize the problem space ℱ of differentiable functions built from a small basis (sin, cos, exp, log, tan, xⁿ) and the operators {+, ×, ∘}. They define an expression tree T_f for each function f and a nesting‑depth metric δ(e) that counts the maximum number of function applications along any root‑to‑leaf path (ignoring pure additive or multiplicative nodes). Based on δ, they partition the task set ℙ into five difficulty levels D₁–D₅, where D₁ contains base‑case derivatives (e.g., d/dx