Parallel Training in Spiking Neural Networks

The bio-inspired integrate-fire-reset mechanism of spiking neurons constitutes the foundation for efficient processing in Spiking Neural Networks (SNNs). Recent progress in large models demands that spiking neurons support highly parallel computation to scale efficiently on modern GPUs. This work proposes a novel functional perspective that provides general guidance for designing parallel spiking neurons. We argue that the reset mechanism, which induces complex temporal dependencies and hinders parallel training, should be removed. However, any such modification should satisfy two principles: 1) preserving the functions of reset as a core biological mechanism; and 2) enabling parallel training without sacrificing the serial inference ability of spiking neurons, which underpins their efficiency at test time. To this end, we identify the functions of the reset and analyze how to reconcile parallel training with serial inference, upon which we propose a dynamic decay spiking neuron. We conduct comprehensive testing of our method in terms of: 1) Training efficiency and extrapolation capability. On 16k-length sequences, we achieve a 25.6x training speedup over the pioneering parallel spiking neuron, and our models trained on 2k-length can stably perform inference on sequences as long as 30k. 2) Generality. We demonstrate the consistent effectiveness of the proposed method across five task categories (image classification, neuromorphic event processing, time-series forecasting, language modeling, and reinforcement learning), three network architectures (spiking CNN/Transformer/SSMs), and two spike activation modes (spike/integer activation). 3) Energy consumption. The spiking firing of our neuron is lower than that of vanilla and existing parallel spiking neurons.

💡 Research Summary

**
The paper tackles a fundamental bottleneck in Spiking Neural Networks (SNNs): the integrate‑fire‑reset (IF‑reset) mechanism that couples temporal dynamics across timesteps and prevents efficient parallel training on modern GPUs. While the reset operation (hard or soft) provides two essential functions—introducing non‑linearity between inputs and membrane potential, and controlling the membrane potential to avoid runaway accumulation—the authors argue that these functions can be realized more flexibly without the strict temporal dependency imposed by reset.

The authors first formalize the conditions under which a spiking neuron can support both parallel training and serial inference: the neuron’s output must depend only on past inputs (causality) while allowing each timestep’s computation to be expressed as a function that can be evaluated independently across the whole sequence. Guided by this functional view, they propose the Dynamic Decay Spiking Neuron (DDSN), which replaces the reset with two key components:

1. Input‑dependent dynamic decay – instead of a fixed decay factor β, the decay coefficient βₜ is computed from the current input (e.g., via a sigmoid of a linear projection). This makes the membrane’s “memory length” adaptive: large inputs cause rapid decay, small inputs allow longer integration, thereby achieving a variable Δ‑short control.

2. Causal convolutional integration – the pre‑spike membrane potential Hₜ is obtained by a 1‑D causal convolution over a finite window of past inputs: Hₜ = Σₖ αₖ·Xₜ₋ₖ, where αₖ are learnable parameters. This formulation aggregates the contributions of all past inputs in a single parallelizable operation, eliminating the need for sequential recurrence during back‑propagation.

3. Smooth surrogate spike function – a differentiable approximation (e.g., sigmoid or swish) is used during training, but the forward pass still yields binary spikes, preserving the binary nature of spiking communication.

Together, these mechanisms reproduce the two reset functions: the adaptive decay provides flexible, input‑driven control over how long a large membrane potential influences future spikes (Δ‑short control), while the bounded convolution ensures the membrane potential remains within a stable range (long‑term control). Importantly, because the convolution is causal, the neuron can be evaluated step‑by‑step at inference time exactly like a traditional LIF neuron, preserving the energy‑efficient, event‑driven nature of SNNs.

Experimental Evaluation
The authors conduct extensive experiments across five task categories—image classification, neuromorphic event processing, time‑series forecasting, language modeling, and reinforcement learning—and three network families—spiking CNNs, spiking Transformers, and spiking State‑Space Models (SSMs). Key findings include:

• Training speed: On sequences of length 16 k, DDSN‑based models achieve a 25.6× speedup over the pioneering Parallel Spiking Neuron (PSN) baseline, thanks to full GPU parallelization of the causal convolution.
• Extrapolation: Models trained on 2 k‑step sequences reliably infer on sequences up to 30 k steps, demonstrating that dynamic decay prevents membrane overflow and maintains temporal discriminability over very long horizons.
• Performance: Across all benchmarks, DDSN matches or slightly exceeds the accuracy of both vanilla LIF and PSN baselines, with notable gains in tasks requiring long‑range dependencies (language modeling, reinforcement learning).
• Energy efficiency: The average spike rate of DDSN neurons is 15–30 % lower than that of vanilla LIF and PSN neurons, implying reduced power consumption on neuromorphic hardware.

Broader Impact
The work shows that removing the reset does not inevitably degrade SNN capabilities; rather, a carefully designed dynamic decay and causal convolution can provide richer, more adaptable non‑linear dynamics while enabling massive parallelism. This opens a practical pathway for scaling SNNs to the size of modern foundation models, bridging the gap between biologically inspired event‑driven computation and the high‑throughput demands of GPU‑based deep learning. The authors suggest that future neuromorphic chips could directly implement the proposed dynamic decay logic, achieving both the low‑power advantages of spiking computation and the scalability required for large‑scale AI workloads.