Quantifying Ranking Instability Across Evaluation Protocol Axes in Gene Regulatory Network Benchmarking

Benchmark rankings are routinely used to justify scientific claims about method quality in gene regulatory network (GRN) inference, yet the stability of these rankings under plausible evaluation protocol choices is rarely examined. We present a systematic diagnostic framework for measuring ranking instability under protocol shift, including decomposition tools that separate base rate effects from discrimination effects. Using existing single cell GRN benchmark outputs across three human tissues and six inference methods, we quantify pairwise reversal rates across four protocol axes: candidate set restriction (16.3 percent, 95 percent CI 11.0 to 23.4 percent), tissue context (19.3 percent), reference network choice (32.1 percent), and symbol mapping policy (0.0 percent). A permutation null confirms that observed reversal rates are far below random order expectations (0.163 versus null mean 0.500), indicating partially stable but non invariant ranking structure. Our decomposition reveals that reversals are driven by changes in the relative discrimination ability of methods rather than by base rate inflation, a finding that challenges a common implicit assumption in GRN benchmarking. We propose concrete reporting practices for stability aware evaluation and provide a diagnostic toolkit for identifying method pairs at risk of reversal.

💡 Research Summary

The paper addresses a critical gap in gene regulatory network (GRN) inference benchmarking: the stability of method rankings under plausible variations of the evaluation protocol. The authors introduce a systematic diagnostic framework that quantifies ranking instability when the protocol shifts, and they decompose observed ranking changes into two components – a base‑rate effect (the overall fraction of positive edges in the candidate set) and a discrimination effect (the relative ability of methods to separate true from false edges within that set).

Using existing single‑cell GRN benchmark outputs from three human tissues (kidney, lung, immune) and six inference methods (including scGPT, GENIE3, GRNBoost2, SCENIC, and random baselines), the authors evaluate four protocol axes: (1) candidate‑set restriction (e.g., all possible gene pairs versus TF‑source‑target pairs), (2) tissue context, (3) reference network choice (DoRothEA, TRRUST, OmniPath, STRING, and unions), and (4) symbol‑mapping policy (different gene identifier resolutions). For each axis they compute pairwise reversal rates – the proportion of method pairs whose relative ranking flips when the protocol changes.

Key empirical findings:

• Candidate‑set restriction yields 22 reversals out of 135 possible pairs (16.3 % ± 11.0–23.4 % CI). The effect is tissue‑dependent, with immune evaluations showing the highest sensitivity (40 % reversal for TF‑source‑target restriction).
• Tissue shifts cause 26 reversals (19.3 % ± 13.5–26.7 % CI). The reversal rate rises with tighter candidate‑set constraints, indicating that more curated edge sets amplify tissue‑specific differences.
• Reference‑network shifts produce the largest instability: 34 reversals out of 106 pairs (32.1 % ± 24.0–41.5 % CI), especially when moving from the Beeline GSD reference to a DoRothEA‑TRRUST union (42.9 % reversal).
• Symbol‑mapping changes (different gene ID normalization) result in zero reversals (0 % ± 0.0–2.3 % CI) despite large coverage increases, suggesting that mapping policies are order‑preserving transformations.

Decomposition analysis (Δ = b·g, where b is the base‑rate and g the discrimination gap) shows that in all reversal cases the discrimination term, not the base‑rate term, opposes the initial margin. The mean |discrimination|/|base‑rate| ratio for reversal rows is 1.54, confirming that changes in relative discrimination power drive rank flips rather than simple inflation of positive rates.

A permutation null experiment (5 000 random shuffles of method scores preserving candidate‑set structure) yields a mean reversal rate of 0.500, whereas the observed candidate‑set reversal rate is 0.163, demonstrating that rankings retain substantial shared structure but are far from invariant.

To aid practitioners, the authors propose an “instability‑region” screening tool. By estimating the maximum observed margin shift (B) across a protocol family, any method pair whose initial margin |Δ₁| ≤ B is flagged as potentially unstable. Leave‑one‑tissue‑out cross‑validation identifies a quantile threshold (0.25) that achieves precision 0.237, recall 0.636, specificity 0.602, and F1 0.346—high recall with manageable false positives.

The discussion emphasizes that ranking instability is structured, not random, and that discrimination changes are the primary driver. Consequently, normalizing metrics for base‑rate differences will not eliminate instability. The authors recommend reporting stability metrics alongside benchmark results, using multiple reference networks, and applying the provided diagnostic toolkit to identify method pairs at risk of reversal before committing to costly biological validation. This work bridges a methodological gap in GRN benchmarking and offers a template for stability‑aware evaluation in other computational biology domains.