A Unified Matrix-Spectral Framework for Stability and Interpretability in Deep Learning

We develop a unified matrix-spectral framework for analyzing stability and interpretability in deep neural networks. Representing networks as data-dependent products of linear operators reveals spectral quantities governing sensitivity to input perturbations, label noise, and training dynamics. We introduce a Global Matrix Stability Index that aggregates spectral information from Jacobians, parameter gradients, Neural Tangent Kernel operators, and loss Hessians into a single stability scale controlling forward sensitivity, attribution robustness, and optimization conditioning. We further show that spectral entropy refines classical operator-norm bounds by capturing typical, rather than purely worst-case, sensitivity. These quantities yield computable diagnostics and stability-oriented regularization principles. Synthetic experiments and controlled studies on MNIST, CIFAR-10, and CIFAR-100 confirm that modest spectral regularization substantially improves attribution stability even when global spectral summaries change little. The results establish a precise connection between spectral concentration and analytic stability, providing practical guidance for robustness-aware model design and training.

💡 Research Summary

The paper proposes a unified matrix‑spectral framework for analyzing both stability and interpretability of deep neural networks. It begins by representing a feed‑forward network as a data‑dependent product of linear maps and diagonal gating matrices, P(x)=W_L D_{L‑1}(x)…D_1(x) W_1, where each D_i(x) contains the derivatives of the activation functions evaluated at the pre‑activations of layer i. This representation holds almost everywhere and makes the input Jacobian, the parameter Jacobian, the Neural Tangent Kernel (NTK) Gram matrix, and the loss Hessian all appear as linear operators derived from the same underlying product.

To quantify spectral properties, the authors introduce two functionals. Spectral concentration S_C^α(A) measures the proportion of total singular‑value mass captured by the top α fraction of singular values, while spectral entropy H_S(A)=−∑_k p_k log p_k (with p_k=σ_k/∑σ) captures how uniformly the singular values are distributed. High entropy indicates a spread‑out spectrum, low entropy indicates concentration on a few directions.

The central contribution is the Global Matrix Stability Index (GMSI):

S(f_θ;μ,ν)=sup_{(x,y)} max{‖J_f(x)‖_2, ‖∇_θ f_θ(x)‖_2, λ_max(K_θ)^{1/2}, λ_max(H_θ(x,y))^{1/2}}.

Here J_f(x) is the input Jacobian, ∇_θ f_θ(x) the parameter Jacobian, K_θ the empirical NTK matrix built from a sample of inputs, and H_θ(x,y) the Hessian of the loss with respect to parameters for a single example. The index aggregates four spectral quantities that respectively control forward sensitivity, parameter‑perturbation sensitivity, conditioning of kernel‑based training dynamics, and curvature of the loss landscape.

Theorem 4.2 shows that a finite GMSI simultaneously guarantees: (i) a global Lipschitz bound on the network’s output, (ii) Lipschitz continuity of any attribution map that is a function of the Jacobian, (iii) a bound on the NTK condition number κ(K)≤C S^2, and (iv) an upper bound on the largest eigenvalue of the Hessian, which in turn yields a step‑size stability condition for gradient descent. Conversely, each of these properties implies a comparable bound on the GMSI, establishing a tight equivalence up to problem‑dependent constants.

Beyond worst‑case operator‑norm bounds, the paper refines sensitivity analysis with spectral entropy. Theorem 5.1 proves that for isotropic input noise δ with E‖δ‖^2=ε^2, the expected squared output deviation satisfies

E_x E_δ ‖f(x+δ)−f(x)‖^2 ≤ K ε^2 exp(H_S(J_f(x))).

Thus, higher entropy (more uniform singular values) reduces typical amplification, whereas a highly concentrated spectrum can cause large average sensitivity even if the operator norm is modest. Analogous results are derived for the NTK: Theorem 5.3 expresses the effect of label perturbations in terms of the NTK eigenvalues, and Corollary 5.4 shows that, at fixed trace, lower entropy permits larger worst‑case label amplification.

The experimental section validates the theory on three fronts. A synthetic benchmark measures empirical sensitivity under controlled perturbations and confirms the predicted entropy‑sensitivity relationship. On MNIST, the authors compute Jacobian spectra, the attribution condition number κ_attr=σ_1/median(σ), and demonstrate a strong inverse correlation between spectral entropy and κ_attr, indicating that more isotropic Jacobians yield more stable attributions. In CIFAR‑10 and CIFAR‑100, modest changes to the GMSI (e.g., through weight decay) produce negligible shifts in the global index, yet applying a spectral‑entropy regularizer dramatically improves attribution robustness, confirming that targeting the distribution of singular values can be more effective than merely shrinking the largest singular value.

Two practical tools are proposed. Spectral Entropy Regularization adds a penalty on the entropy of weight matrices or the end‑to‑end Jacobian during training, encouraging the optimizer to spread singular‑value mass and thereby improve stability. This complements traditional L2 or spectral‑norm regularization, which only control the maximum singular value. Layer Sensitivity Maps visualize how each layer’s contribution to the overall spectrum changes during training, enabling targeted interventions such as selective scaling, low‑rank projection, or constrained updates on layers that dominate the instability.

Finally, the authors define **Spectral Entropy of the Random Regime (SERR)=E_W