MathlibLemma: Folklore Lemma Generation and Benchmark for Formal Mathematics

While the ecosystem of Lean and Mathlib has enjoyed celebrated success in formal mathematical reasoning with the help of large language models (LLMs), the absence of many folklore lemmas in Mathlib remains a persistent barrier that limits Lean’s usability as an everyday tool for mathematicians like LaTeX or Maple. To address this, we introduce MathlibLemma, the first LLM-based multi-agent system to automate the discovery and formalization of mathematical folklore lemmas. This framework constitutes our primary contribution, proactively mining the missing connective tissue of mathematics. Its efficacy is demonstrated by the production of a verified library of folklore lemmas, a subset of which has already been formally merged into the latest build of Mathlib, thereby validating the system’s real-world utility and alignment with expert standards. Leveraging this pipeline, we further construct the MathlibLemma benchmark, a suite of 4,028 type-checked Lean statements spanning a broad range of mathematical domains. By transforming the role of LLMs from passive consumers to active contributors, this work establishes a constructive methodology for the self-evolution of formal mathematical libraries.

💡 Research Summary

The paper introduces MathlibLemma, a novel multi‑agent framework that automatically discovers, formalizes, and proves “folklore lemmas” – the small, often‑used facts that are missing from the Lean 4 library Mathlib. These lemmas constitute a “last‑mile” bottleneck: they are obvious to human mathematicians but their absence forces both human users and LLM‑based proof assistants to reconstruct routine steps, inflating token usage, increasing hallucination risk, and slowing down proof search.

Framework Overview
MathlibLemma consists of four specialized LLM agents arranged in a pipeline:

1. Discovery Agent – Takes a seed Mathlib file (including its imports) as context and generates a diverse set of candidate lemmas expressed in Lean syntax, each ending with a sorry placeholder. The prompt encourages the model to spot structural gaps, cross‑topic connections, and “obvious” intermediate results that are not yet present in the library.

2. Judge Agent – Implements an “LLM‑as‑a‑judge” step. For each candidate, it evaluates mathematical plausibility while deliberately ignoring syntactic correctness. The output is a binary verdict (correct vs. wrong). This early semantic filter prevents the downstream prover from wasting effort on false statements.

3. Formalizer Agent – Interacts with a Lean server to repair syntax and type errors, insert missing imports, and ensure that the Lean declaration type‑checks. After this stage every candidate is a well‑formed Lean term, though still lacking a proof.

4. Prover Agent – Attempts to construct a proof automatically. It combines traditional Lean tactics (e.g., aesop, simp, linarith) with LLM‑generated proof scripts. Successful proofs are type‑checked by the kernel, yielding a fully verified lemma.

By factorizing the three dominant failure modes—semantic hallucination, syntactic/type errors, and proof search failure—into orthogonal modules, the system achieves higher overall success rates than end‑to‑end generation pipelines.

Empirical Results

• The pipeline produced a verified library of 1 812 folklore lemmas that type‑check in Mathlib.
• 45 % of these lemmas were automatically proved by the Prover Agent; the remaining lemmas are still useful as statements for future work.
• A stratified human audit of the 4 028 unproved residual statements showed that 78 % are mathematically sound, confirming that the Judge and Formalizer stages effectively suppress hallucinations.
• Several generated lemmas have already been merged into the official Mathlib release, demonstrating real‑world impact.

MathlibLemma Benchmark
The authors also release the MathlibLemma benchmark, a collection of 4 028 type‑checked Lean statements spanning algebra, analysis, combinatorics, probability, and more. Unlike traditional benchmarks (MiniF2F, LeanDojo, etc.) that focus on solving isolated Olympiad‑level problems, this benchmark evaluates a model’s ability to expand library coverage by supplying routine background facts. The benchmark is deliberately “saturated”: many statements are already provable by the current system, indicating that the benchmark itself serves as a solution to the last‑mile problem rather than merely a test.

Related Work Positioning
The paper situates MathlibLemma among three research strands: (1) lemma synthesis and library expansion (e.g., LeanConjecturer, Lemmanaid), (2) feedback‑driven repair loops that use compiler errors (Delta Prover, APOLLO), and (3) formal reasoning benchmarks. MathlibLemma differentiates itself by targeting folklore mining—the systematic identification of missing connective tissue—rather than random conjecture generation, and by producing reusable, general‑purpose lemmas instead of problem‑specific scaffolding.

Conclusions and Outlook
MathlibLemma demonstrates that LLMs can move from passive consumers of formal libraries to active contributors that autonomously fill gaps in mathematical knowledge bases. By automating the discovery and verification of folklore lemmas, the framework reduces the friction that currently deters mathematicians from adopting proof assistants for everyday work. The released benchmark provides a new “breadth‑oriented” evaluation metric, encouraging future research to focus on library growth and robustness rather than solely on solving ever‑harder isolated problems. The authors anticipate that continued scaling of multi‑agent pipelines, richer feedback from Lean’s kernel, and tighter integration with community review processes will further accelerate the self‑evolution of formal mathematics ecosystems.