Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning

Multi-task learning (MTL) aims to enable a single model to solve multiple tasks efficiently; however, current parameter-efficient fine-tuning (PEFT) methods remain largely limited to single-task adaptation. We introduce \textbf{Free Sinewich}, a parameter-efficient multi-task learning framework that enables near-zero-cost weight modulation via frequency switching (\textbf{Free}). Specifically, a \textbf{Sine-AWB (Sinewich)} layer combines low-rank factors and convolutional priors into a single kernel, which is then modulated elementwise by a sinusoidal transformation to produce task-specialized weights. A lightweight Clock Net is introduced to produce bounded frequencies that stabilize this modulation during training. Theoretically, sine modulation enhances the rank of low-rank adapters, while frequency separation decorrelates the weights of different tasks. On dense prediction benchmarks, Free Sinewich achieves state-of-the-art performance-efficiency trade-offs (e.g., up to +5.39% improvement over single-task fine-tuning with only 6.53M trainable parameters), offering a compact and scalable paradigm based on frequency-based parameter sharing. Project page: \href{https://casperliuliuliu.github.io/projects/Free-Sinewich/}{https://casperliuliuliu.github.io/projects/Free-Sinewich}.

💡 Research Summary

Free Sinewich introduces a novel parameter‑efficient multi‑task learning (PEFT‑MTL) framework that leverages frequency‑switching to reuse a single set of weights across many tasks while still providing task‑specific behavior. The method builds on low‑rank LoRA adapters but augments them with a convolutional kernel inserted between the low‑rank factors, forming an “A‑W‑B” (Sine‑AWB) module. This fused kernel can be expressed as a block‑Toeplitz‑with‑Toeplitz‑blocks matrix, allowing the three operations (linear reduction, spatial convolution, linear expansion) to be collapsed into a single convolutional layer, which saves computation and memory.

The core novelty lies in applying an element‑wise sine transformation to the fused kernel. For each task t a scalar frequency ω_t is generated by a lightweight Clock Net (LCN). The LCN receives a learnable task token p_t, passes it through a single‑layer MLP, and outputs ω_t = s·(tanh(W_q·ReLU(p_t)) + c), where s and c are learnable scale and offset parameters. The sine mapping M_t = sin(ω_t·M_AWB) dramatically increases the effective rank of the originally low‑rank adapter, providing richer representations without adding trainable parameters.

Because sine is a non‑linear operator, the transformed matrices can contain high‑frequency noise. To mitigate this, the authors apply a 7×7 Gaussian low‑pass filter (σ = 1) to obtain a smoothed kernel eM_t, which preserves structural details while suppressing spurious oscillations and stabilizing training.

In the decoder, Free Sinewich departs from prior PEFT‑MTL approaches that replicate an entire decoder per task (leading to O(T) parameter growth). Instead, a shared decoder group Ψ_shd reuses the same convolutional weights M_AWB across all tasks. Each task’s forward pass multiplies the shared kernel by its task‑specific frequency and sine transformation, yielding a task‑specific convolution eM_t * x_t + b_t. The remaining layers (BN, ReLU, final conv) are shared, dramatically reducing the overall parameter count while still allowing each task to extract its own features.

The authors provide two theoretical arguments: (1) sine modulation raises the effective rank of low‑rank adapters, thereby enhancing expressive power; (2) distinct frequencies generate nearly orthogonal nonlinear mappings, decorrelating task‑specific weight updates and alleviating gradient interference.

Experiments on two dense‑prediction benchmarks—Pascal‑Context (semantic segmentation, human parts, saliency, surface normals) and NYUD‑v2 (semantic segmentation, depth, normals, edge detection)—show that Free Sinewich achieves state‑of‑the‑art performance‑efficiency trade‑offs. With only 6.53 M trainable parameters (well under 1 % of the full backbone), it outperforms single‑task fine‑tuning by up to +5.39 % average relative improvement and surpasses recent PEFT‑MTL baselines such as MTLoRA, DiT‑Former, and DIT‑Task across all metrics (mIoU, RMSE, ODSF). The average relative improvement metric Δm confirms that the method consistently benefits every task while keeping the parameter budget minimal.

Limitations include the focus on vision dense‑prediction tasks; applicability to NLP, speech, or time‑series domains remains to be explored. The scalar frequency design may be insufficient for highly complex task relationships, suggesting future work on multi‑dimensional or complex‑valued frequency embeddings. Moreover, while the Clock Net guarantees bounded frequencies, a deeper analysis of the learned frequency distribution and its dynamics during training would strengthen the empirical claims.

In summary, Free Sinewich offers a compelling synthesis of low‑rank adaptation, convolutional priors, sinusoidal modulation, and lightweight frequency generation to achieve truly parameter‑shared multi‑task learning. By mimicking the brain’s oscillatory multiplexing, it establishes a new paradigm for efficient, scalable, and conflict‑free multi‑task deep learning.