Robust Machine Learning for Regulatory Sequence Modeling under Biological and Technical Distribution Shifts

Robust machine learning for regulatory genomics is studied under biologically and technically induced distribution shifts. Deep convolutional and attention based models achieve strong in distribution performance on DNA regulatory sequence prediction tasks but are usually evaluated under i.i.d. assumptions, even though real applications involve cell type specific programs, evolutionary turnover, assay protocol changes, and sequencing artifacts. We introduce a robustness framework that combines a mechanistic simulation benchmark with real data analysis on a massively parallel reporter assay (MPRA) dataset to quantify performance degradation, calibration failures, and uncertainty based reliability. In simulation, motif driven regulatory outputs are generated with cell type specific programs, PWM perturbations, GC bias, depth variation, batch effects, and heteroscedastic noise, and CNN, BiLSTM, and transformer models are evaluated. Models remain accurate and reasonably calibrated under mild GC content shifts but show higher error, severe variance miscalibration, and coverage collapse under motif effect rewiring and noise dominated regimes, revealing robustness gaps invisible to standard i.i.d. evaluation. Adding simple biological structural priors motif derived features in simulation and global GC content in MPRA improves in distribution error and yields consistent robustness gains under biologically meaningful genomic shifts, while providing only limited protection against strong assay noise. Uncertainty-aware selective prediction offers an additional safety layer that risk coverage analyses on simulated and MPRA data show that filtering low confidence inputs recovers low risk subsets, including under GC-based out-of-distribution conditions, although reliability gains diminish when noise dominates.

💡 Research Summary

This paper investigates how modern deep learning models for regulatory DNA sequence prediction—specifically convolutional neural networks (CNNs), bidirectional LSTMs (BiLSTMs), and Transformers—perform when faced with biologically and technically induced distribution shifts. While these models achieve high accuracy under the usual i.i.d. assumption, real‑world applications involve cell‑type‑specific transcription factor programs, evolutionary turnover of motifs, changes in assay protocols, sequencing depth variations, batch effects, and GC‑bias. To quantify robustness, the authors construct a two‑pronged evaluation framework.

First, they develop a mechanistic simulation benchmark. Synthetic 1 kb sequences are generated with a controllable GC‑content parameter. A set of position weight matrices (PWMs) represents transcription factor motifs; motif occurrences contribute additively to a latent regulatory signal. Cell‑type‑specific activity coefficients modulate each motif’s contribution, producing cell‑type‑specific outputs. Technical shifts are introduced independently: multiplicative log‑normal depth scaling, additive Gaussian batch offsets, and a GC‑dependent scaling factor. Heteroscedastic Gaussian noise is added to model assay‑specific variance. Biological OOD scenarios are simulated by (i) sampling new TF activity coefficients for test cell types, (ii) perturbing PWMs (δ‑probability substitution) to mimic evolutionary motif turnover, and (iii) performing structured perturbations such as motif knockout, insertion, rewiring, and masking.

Second, the authors evaluate models on a real massively parallel reporter assay (MPRA) dataset. They augment the raw DNA input with two structural priors: (a) global GC‑content and (b) motif activation scores derived from PWM scanning. A hybrid architecture concatenates a CNN‑derived sequence embedding with a small MLP‑processed motif vector, followed by a heteroscedastic Gaussian head that predicts both mean activity and variance.

Performance is measured along three research questions. RQ1 assesses in‑distribution (ID) versus out‑of‑distribution (OOD) accuracy, expected calibration error (ECE), negative log‑likelihood, and coverage. Under mild GC‑shifts, all models retain low mean‑squared error (MSE ≈ 0.10) and modest Var‑ECE (≈ 0.04). However, motif‑effect rewiring doubles the error (MSE ≈ 0.21), reduces 90 % coverage to 0.66, and inflates Var‑ECE to 0.34. Heteroscedastic noise degrades performance dramatically (MSE ≈ 1.5, Var‑ECE ≈ 1.4, coverage ≈ 0.30). The combination of both shifts yields the worst degradation (MSE ≈ 1.63, Var‑ECE ≈ 1.51). Binary classifiers exhibit analogous drops, with accuracies approaching chance and large calibration errors.

RQ2 examines whether structural priors improve robustness. Adding motif scores and GC‑content reduces ID MSE to ≈ 0.08 and mitigates OOD error growth by roughly 30 %. Calibration also improves, but the benefit diminishes when technical noise dominates, indicating that priors cannot fully compensate for severe assay artifacts.

RQ3 explores uncertainty‑aware selective prediction. The authors compute predictive variance via MC‑Dropout, deep ensembles, and the heteroscedastic head. By abstaining on high‑uncertainty inputs, risk‑coverage curves shift leftward, showing that low‑risk subsets can be recovered, especially under mild GC‑shifts (≈ 20 % of samples filtered yields a 40 % reduction in overall error). In noise‑dominated regimes, the gains are modest, reflecting the limits of uncertainty estimation when the signal‑to‑noise ratio is low.

Overall, the study reveals that current regulatory sequence models are robust to shallow covariate shifts but are vulnerable to mechanistic (concept) shifts and heteroscedastic assay noise—failure modes invisible to standard i.i.d. benchmarks. Incorporating biologically motivated structural priors and leveraging uncertainty for selective prediction provide practical mitigation strategies, though they do not fully resolve robustness gaps under extreme technical perturbations. The work establishes a reproducible simulation platform and a concrete evaluation pipeline that can guide future development of more resilient genomic deep learning models.