CHIMERA-Bench: A Benchmark Dataset for Epitope-Specific Antibody Design

Computational antibody design has seen rapid methodological progress, with dozens of deep generative methods proposed in the past three years, yet the field lacks a standardized benchmark for fair comparison and model development. These methods are evaluated on different SAbDab snapshots, non-overlapping test sets, and incompatible metrics, and the literature fragments the design problem into numerous sub-tasks with no common definition. We introduce \textsc{Chimera-Bench} (\textbf{C}DR \textbf{M}odeling with \textbf{E}pitope-guided \textbf{R}edesign), a unified benchmark built around a single canonical task: \emph{epitope-conditioned CDR sequence-structure co-design}. \textsc{Chimera-Bench} provides (1) a curated, deduplicated dataset of \textbf{2,922} antibody-antigen complexes with epitope and paratope annotations; (2) three biologically motivated splits testing generalization to unseen epitopes, unseen antigen folds, and prospective temporal targets; and (3) a comprehensive evaluation protocol with five metric groups including novel epitope-specificity measures. We benchmark representative methods spanning different generative paradigms and report results across all splits. \textsc{Chimera-Bench} is the largest dataset of its kind for the antibody design problem, allowing the community to develop and test novel methods and evaluate their generalizability. The source code and data are available at: https://github.com/mansoor181/chimera-bench.git

💡 Research Summary

The paper introduces CHIMERA‑Bench, a unified benchmark for epitope‑conditioned antibody CDR design that addresses the fragmented evaluation landscape of recent deep generative methods. Existing works are trained on different snapshots of the Structural Antibody Database (SAbDab), use non‑overlapping test sets, and report incompatible metrics (varying contact cut‑offs, inconsistent RMSD calculations). Moreover, the design problem has been split into many sub‑tasks (inverse folding, docking, affinity optimization, de‑novo generation, etc.), making direct comparison impossible.

CHIMERA‑Bench consolidates these disparate efforts into a single canonical task: given an antigen structure, a specified epitope, and an antibody framework, generate CDR residues (sequence and 3‑D coordinates) that are structurally plausible, contact the target epitope, and avoid off‑target binding. The benchmark provides (1) a curated, deduplicated dataset of 2,922 high‑quality antibody‑antigen complexes drawn from SAbDab, each annotated with IMGT and Chothia numbering, CDR masks, and epitope/paratope residues defined by a 4.5 Å contact distance; (2) three biologically motivated data splits—epitope‑group (novel epitope patterns), antigen‑fold (unseen antigen structures), and temporal (future PDB entries)—all enforced at the cluster level to prevent leakage; and (3) a comprehensive evaluation protocol comprising five metric groups: sequence quality (AAR, CAAR, perplexity), structural accuracy (Cα RMSD, TM‑score), interface quality (Fnat, iRMSD, DockQ), epitope specificity (precision, recall, F1 on epitope contacts), and designability (count of known manufacturing liability motifs). The epitope‑specificity metrics are novel to the field and directly assess whether generated CDRs bind the intended site.

The authors re‑train eleven representative antibody design methods spanning six generative paradigms (equivariant GNNs, diffusion models, flow‑matching, autoregressive, hierarchical equivariant networks, and conjoined ODEs) using the same codebases and default hyper‑parameters on the CHIMERA‑Bench training split. Results on the primary epitope‑group test split show that equivariant GNNs (RAAD, MEAN, dyMEAN) achieve the highest amino‑acid recovery (AAR ≈ 0.37) but relatively low epitope F1 (≈ 0.10). Diffusion‑based models (DiffAb, AbFlowNet, AbMEGD) recover fewer native residues (AAR ≈ 0.21) yet attain higher epitope F1 (≈ 0.20), indicating better targeting of the specified epitope. dyMEAN uniquely balances both aspects, delivering AAR = 0.37 and epitope F1 = 0.23, as well as the best native contact recovery (Fnat) among all methods. Some models (AbDockGen, AbODE) produce low RMSD CDR loops but suffer from large iRMSD and low DockQ, revealing that locally correct loops can be misplaced relative to the antigen surface.

Across the three splits, most methods display stable performance, though modest drops are observed on the temporal split, confirming that generalization to truly novel antigens remains challenging. The benchmark also highlights that current designs recover only a small fraction of native contacts (Fnat ≈ 0.02–0.05), yet maintain moderate DockQ scores because redesign is limited to a single CDR, preserving overall interface geometry.

In conclusion, CHIMERA‑Bench delivers a rigorously curated dataset, biologically realistic data partitions, and a multi‑facet evaluation suite that together enable fair, reproducible comparison of antibody design algorithms. By defining a single, therapeutically relevant task—epitope‑conditioned CDR co‑design—it unifies previously fragmented research and provides a solid foundation for future advances in generative antibody engineering.