Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks

Task-trained recurrent neural networks (RNNs) are widely used in neuroscience and machine learning to model dynamical computations. To gain mechanistic insight into how neural systems solve tasks, prior work often reverse-engineers individual trained networks. However, different RNNs trained on the same task and achieving similar performance can exhibit strikingly different internal solutions, a phenomenon known as solution degeneracy. Here, we develop a unified framework to systematically quantify and control solution degeneracy across three levels: behavior, neural dynamics, and weight space. We apply this framework to 3,400 RNNs trained on four neuroscience-relevant tasks: flip-flop memory, sine wave generation, delayed discrimination, and path integration, while systematically varying task complexity, learning regime, network size, and regularization. We find that higher task complexity and stronger feature learning reduce degeneracy in neural dynamics but increase it in weight space, with mixed effects on behavior. In contrast, larger networks and structural regularization reduce degeneracy at all three levels. These findings empirically validate the Contravariance Principle and provide practical guidance for researchers seeking to tune the variability of RNN solutions, either to uncover shared neural mechanisms or to model the individual variability observed in biological systems. This work provides a principled framework for quantifying and controlling solution degeneracy in task-trained RNNs, offering new tools for building more interpretable and biologically grounded models of neural computation.

💡 Research Summary

This paper tackles the often‑overlooked issue that recurrent neural networks (RNNs) trained on the same task can converge to markedly different internal solutions—a phenomenon termed “solution degeneracy.” The authors introduce a unified framework that quantifies degeneracy at three complementary levels: (1) behavior, measured as the variability of out‑of‑distribution (OOD) performance across independently trained networks; (2) neural dynamics, assessed with Dynamical Similarity Analysis (DSA), which compares the linear next‑step operators inferred from each network’s hidden‑state trajectories; and (3) weight space, measured by a permutation‑invariant Frobenius distance (d_PIF) between recurrent weight matrices.

To explore what factors shape degeneracy, the authors conduct a massive empirical sweep: 3,400 RNNs (50 random seeds per condition) are trained on four neuroscience‑inspired tasks—N‑bit Flip‑Flop, Delayed Discrimination, Sine‑Wave Generation, and Path Integration. For each task they systematically vary four key dimensions: (i) task complexity (by increasing the number of independent input‑output channels), (ii) learning regime/feature learning strength (adding auxiliary losses), (iii) network size (hidden‑unit width), and (iv) structural regularization (L1, sparsity, low‑rank constraints). All networks are vanilla discrete‑time tanh RNNs trained with BPTT and Adam until a near‑asymptotic loss threshold is reached, ensuring comparable final performance before degeneracy analysis.

Key findings:

• Task complexity exerts a contravariant effect. As the number of channels grows, DSA distances shrink—networks develop more similar trajectories, supporting the Contravariance Principle that harder tasks constrain the space of viable dynamical solutions. However, weight‑space distances (d_PIF) increase with complexity, indicating that harder tasks generate more isolated minima in the loss landscape. Behavioral degeneracy (σ_OOD) generally declines with complexity, but the relationship is not uniform across tasks.

• Feature‑learning strength mirrors the complexity effect: stronger auxiliary objectives reduce dynamical degeneracy while inflating weight degeneracy. This suggests that forcing the network to learn richer internal representations narrows the set of admissible dynamics but pushes the optimizer into more diverse regions of weight space.

• Network size produces covariant reductions across all three levels. Larger hidden layers make the optimization landscape smoother, allowing different initializations to converge to similar dynamics, weights, and OOD behavior.

• Structural regularization (e.g., L1 sparsity) also yields covariant reductions, confirming that explicit constraints on connectivity promote consistency across independently trained models.

The authors further examine representational degeneracy using SVCCA and find that it does not track dynamical or weight degeneracy, underscoring that identical behavioral performance can arise from distinct internal representations.

Practical implications are distilled into a guidance table (Table 1) that tells researchers how to tune degeneracy depending on their scientific goal: to uncover shared neural mechanisms one should increase task complexity, enlarge the network, and apply regularization; to model biological variability one should lower complexity, use smaller networks, and relax regularization.

Overall, the paper delivers a rigorous, multi‑level metric suite, a large‑scale empirical validation of the Contravariance Principle, and actionable recommendations for controlling solution variability in task‑trained RNNs—advancing both interpretability in computational neuroscience and robustness in machine‑learning applications.