Target noise: A pre-training based neural network initialization for efficient high resolution learning

Weight initialization plays a crucial role in the optimization behavior and convergence efficiency of neural networks. Most existing initialization methods, such as Xavier and Kaiming initializations, rely on random sampling and do not exploit information from the optimization process itself. We propose a simple, yet effective, initialization strategy based on self-supervised pre-training using random noise as the target. Instead of directly training the network from random weights, we first pre-train it to fit random noise, which leads to a structured and non-random parameter configuration. We show that this noise-driven pre-training significantly improves convergence speed in subsequent tasks, without requiring additional data or changes to the network architecture. The proposed method is particularly effective for implicit neural representations (INRs) and Deep Image Prior (DIP)-style networks, which are known to exhibit a strong low-frequency bias during optimization. After noise-based pre-training, the network is able to capture high-frequency components much earlier in training, leading to faster and more stable convergence. Although random noise contains no semantic information, it serves as an effective self-supervised signal (considering its white spectrum nature) for shaping the initialization of neural networks. Overall, this work demonstrates that noise-based pre-training offers a lightweight and general alternative to traditional random initialization, enabling more efficient optimization of deep neural networks.

💡 Research Summary

The paper introduces a remarkably simple yet effective strategy for neural network weight initialization: pre‑training the network to fit random white‑noise targets before tackling the actual downstream task. Traditional initializations such as Xavier or Kaiming rely on random sampling and aim only to preserve variance across layers; they do not exploit any information about the network’s own optimization dynamics. In contrast, the proposed “target‑noise” pre‑training replaces the true labels with a noise image (Gaussian or uniform) and runs a short self‑supervised optimization phase (typically a few hundred iterations). This process moves the parameters from a purely random point to a structured state that reflects the architecture’s inductive biases without using any semantic data.

The authors ground their empirical observations in Neural Tangent Kernel (NTK) theory. By expanding the network output around the initialization, the NTK captures how gradients propagate during training. With standard random initialization, the NTK eigenvectors exhibit a clear frequency hierarchy: low‑frequency modes dominate the leading eigenvectors, and the corresponding eigenvalues decay rapidly, which explains the well‑known spectral bias of implicit neural representations (INRs) and Deep Image Prior (DIP) networks—early training captures coarse structures while fine details appear only later. After noise‑driven pre‑training, the NTK spectrum becomes markedly flatter: eigenvectors lose their ordered frequency pattern and resemble random projections, while eigenvalues decay much more slowly. Consequently, gradient energy is distributed more evenly across modes, allowing high‑frequency components to be learned much earlier. Visualizations of the NTK matrix also show a narrower bright diagonal band, indicating increased locality and higher sensitivity to fine‑scale variations.

Extensive experiments validate these theoretical insights. In a coordinate‑based image representation task (SIREN MLP mapping 2‑D coordinates to RGB values), the noise‑pre‑trained model, despite starting with a higher loss (because it first fits noise), surpasses the standard‑initialized model within the first 100 iterations and maintains a lead throughout training. PSNR curves confirm faster convergence and higher final quality. Similar gains are observed on DIP‑style tasks: single‑image super‑resolution, denoising, and inpainting. After only 50 training steps, the noise‑pre‑trained networks already produce sharper, more detailed reconstructions, whereas the baseline networks remain overly smooth. Across different architectures (MLP, CNN) and hyper‑parameter settings, the method consistently reduces early‑stage loss oscillations and accelerates overall training.

The paper highlights several practical advantages. No extra labeled data or architectural modifications are required; the pre‑training cost is negligible compared to the full training budget; and the approach is agnostic to the downstream task, making it a drop‑in replacement for any random initialization. Potential drawbacks are also discussed. The increased emphasis on high‑frequency modes can make the network more sensitive to small perturbations, possibly reducing robustness. Moreover, the NTK analysis is exact only in the infinite‑width limit, so the observed benefits in finite‑width networks may vary with depth, width, or activation function. Future work could explore optimal noise distributions, adaptive pre‑training durations, combinations with other initialization tricks (e.g., orthogonal weights), and integration into large‑scale self‑supervised pre‑training pipelines.

In summary, the study demonstrates that fitting white‑noise—a task devoid of semantic meaning—serves as a powerful self‑supervised signal for shaping neural network initializations. By flattening the NTK spectrum and mitigating spectral bias, noise‑based pre‑training yields faster convergence, earlier recovery of high‑frequency details, and more stable early optimization across a range of image‑centric tasks, offering a lightweight and broadly applicable alternative to conventional random weight initialization.