Asymptotic Smoothing of the Lipschitz Loss Landscape in Overparameterized One-Hidden-Layer ReLU Networks

We study the topology of the loss landscape of one-hidden-layer ReLU networks under overparameterization. On the theory side, we (i) prove that for convex $L$-Lipschitz losses with an $\ell_1$-regularized second layer, every pair of models at the same loss level can be connected by a continuous path within an arbitrarily small loss increase $ε$ (extending a known result for the quadratic loss); (ii) obtain an asymptotic upper bound on the energy gap $ε$ between local and global minima that vanishes as the width $m$ grows, implying that the landscape flattens and sublevel sets become connected in the limit. Empirically, on a synthetic Moons dataset and on the Wisconsin Breast Cancer dataset, we measure pairwise energy gaps via Dynamic String Sampling (DSS) and find that wider networks exhibit smaller gaps; in particular, a permutation test on the maximum gap yields $p_{perm}=0$, indicating a clear reduction in the barrier height.

💡 Research Summary

The paper investigates the loss‑landscape topology of one‑hidden‑layer ReLU networks when the hidden layer is heavily over‑parameterized. The authors consider a model Φ(x;W₁,θ)=θᵀ·Z(W₁x) where W₁∈ℝ^{m×n} contains unit‑norm rows and Z denotes element‑wise ReLU. The training objective is a convex, L‑Lipschitz loss ℓ(Y,·) applied to the network logits, plus an ℓ₁ regularization term κ‖θ‖₁ on the output weights:
F₀(W₁,θ)=E_{(X,Y)∼P}