BABE: Biology Arena BEnchmark

The rapid evolution of large language models (LLMs) has expanded their capabilities from basic dialogue to advanced scientific reasoning. However, existing benchmarks in biology often fail to assess a critical skill required of researchers: the ability to integrate experimental results with contextual knowledge to derive meaningful conclusions. To address this gap, we introduce BABE(Biology Arena BEnchmark), a comprehensive benchmark designed to evaluate the experimental reasoning capabilities of biological AI systems. BABE is uniquely constructed from peer-reviewed research papers and real-world biological studies, ensuring that tasks reflect the complexity and interdisciplinary nature of actual scientific inquiry. BABE challenges models to perform causal reasoning and cross-scale inference. Our benchmark provides a robust framework for assessing how well AI systems can reason like practicing scientists, offering a more authentic measure of their potential to contribute to biological research.

💡 Research Summary

The paper introduces BABE (Biology Arena Benchmark), a novel evaluation suite designed to measure the experimental reasoning abilities of large language models (LLMs) in the biological sciences. Existing benchmarks largely focus on isolated tasks such as sequence classification or structural prediction and rarely require models to integrate experimental results with contextual knowledge—a skill essential for real‑world biological research. BABE addresses this gap by constructing its items directly from peer‑reviewed papers and real experimental studies, ensuring that each question reflects the complexity, multimodality, and interdisciplinary nature of contemporary biology.

Each benchmark instance consists of a triplet of questions (Q₁, Q₂, Q₃) derived from a single source document. The logical relationship between consecutive questions is explicitly labeled as either Strong Correlation (Rₛₜᵣₒₙg) or Weak Correlation (R_wₑₐₖ). In a strong‑correlation set, the answer to Qᵢ is required to solve Qᵢ₊₁, thereby testing multi‑step causal reasoning and chain‑of‑thought capabilities. In a weak‑correlation set, the questions are independent, probing the model’s ability to maintain multiple contexts and extract parallel pieces of information. This dual structure enables a fine‑grained diagnosis of both sequential inference depth and breadth of contextual retrieval.

The data collection pipeline proceeds through four stages: (1) curating a corpus of recent, high‑impact biological publications across twelve sub‑fields (cell biology, plant science, neuroscience, etc.); (2) having domain experts author three questions per source that demand conceptual understanding, methodological interpretation, and higher‑order reasoning; (3) a second‑round expert review that assigns the strong/weak correlation label, verifies factual correctness, and ensures self‑containment; (4) a refinement step aided by LLMs to filter out trivial or overly simplistic items. Importantly, the benchmark retains the original multimodal evidence—Western blot images, electrophoresis gels, quantitative plots—so models must process visual and numeric data alongside text.

In the experimental evaluation, twelve state‑of‑the‑art LLMs are tested on BABE. OpenAI’s GPT‑5.1‑high achieves the highest overall average score of 52.31, with balanced performance on both strong (51.79) and weak (52.86) correlation subsets, indicating robust reasoning that generalizes across dependency structures. Gemini‑3‑Pro‑Preview‑Exp excels in the weak‑correlation condition (55.16) but lags in strong correlation (49.05), suggesting a strength in contextual synthesis but a weakness in chained inference. Other models such as Claude‑Sonnet‑4.5‑thinking‑azure and Gemini‑2.5‑Pro display relatively stable scores across both subsets, reflecting more consistent reasoning strategies. Low‑performing models (e.g., GLM‑4.5‑V) score below 25 on average, highlighting substantial gaps in handling the benchmark’s demands.

A deeper behavioral analysis classifies inference steps into “Deep Reasoning,” “Self‑Exploration,” and “Self‑Reflection.” High‑scoring models allocate a significantly larger proportion of their inference steps to Deep Reasoning, whereas lower‑scoring models rely more on shallow pattern matching. This evidence supports the authors’ claim that BABE rewards genuine logical processing rather than surface‑level memorization.

The paper’s contributions are threefold: (1) a question design that forces simultaneous interpretation of experimental data and domain context, effectively recreating real research scenarios; (2) an explicit strong/weak correlation taxonomy that isolates multi‑hop causal reasoning from parallel information extraction; (3) a multimodal, research‑derived dataset that pushes models beyond text‑only benchmarks. The authors argue that BABE provides a more authentic measure of an AI system’s potential to assist in hypothesis generation, experimental design, and result interpretation in biology.

Future work outlined includes expanding the benchmark to cover additional experimental modalities (e.g., CRISPR editing outcomes, large‑scale omics datasets), integrating human‑AI collaborative workflows, and evaluating multimodal models that jointly process text, images, and numeric tables. By establishing a rigorous, research‑grounded standard, BABE aims to become a cornerstone for the development and assessment of biologically‑savvy AI systems.