Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages

Recent speech foundation models excel at multilingual automatic speech recognition (ASR) for high-resource languages, but adapting them to low-resource languages remains challenging due to data scarcity and efficiency constraints. Full-model fine-tuning is computationally expensive and prone to overfitting, while parameter-efficient methods like LoRA apply adaptation uniformly across layers, overlooking internal representations thus compromising effectiveness and efficiency. We analyze multilingual ASR models and reveal a U-shaped adaptability pattern: early and late layers are language-specific and require more adaptation, while intermediate layers retain shared semantics and need less. Building on this observation, we propose DAMA, a Depth-Aware Model Adaptation framework that allocates adaptation capacity according to each layer’s role. DAMA also introduces Singular Value Decomposition (SVD)-based initialization to constrain adaptation and preserve the U-shaped pattern, as well as a frozen middle-layer basis for further efficiency. Evaluated on 18 low-resource languages across two benchmark datasets, DAMA matches or surpasses state-of-the-art accuracy with 80% fewer trainable parameters, achieves a 29% error reduction under extreme data scarcity, and significantly improves memory, training time, and computational efficiency over baselines. These results highlight the benefits of structure-aware adaptation for efficient, scalable multilingual ASR.

💡 Research Summary

**
The paper addresses the challenge of adapting large multilingual speech‑recognition foundations to low‑resource languages. While full‑parameter fine‑tuning is costly and prone to over‑fitting, existing parameter‑efficient methods such as LoRA treat every layer uniformly, ignoring the internal structure of language representations. By probing each decoder layer with a linear language‑identification classifier, the authors discover a pronounced U‑shaped plasticity: early and late layers encode language‑specific acoustic and lexical cues, whereas middle layers contain language‑agnostic semantic features. Full fine‑tuning collapses this structure, forcing middle layers to become language‑specific and dramatically hurting performance on high‑resource languages.

Motivated by this insight, the authors propose Depth‑Aware Model Adaptation (DAMA). DAMA introduces three mechanisms: (1) a Depth‑Aware Rank Schedule that assigns higher LoRA ranks to early and late layers while keeping a minimal rank for middle layers, mirroring the U‑shape; (2) SVD‑based initialization for the middle‑layer adapters, aligning the low‑rank updates orthogonal to the dominant singular vectors of the frozen weights, thereby preserving the shared semantic subspace; and (3) Basis‑Protected Projection, which freezes a subset of middle‑layer parameters to further reduce trainable parameters and improve stability in extreme low‑data regimes.

Experiments on 18 low‑resource languages from the Common Voice and FLEURS benchmarks show that DAMA matches or exceeds state‑of‑the‑art accuracy while cutting trainable parameters by roughly 80 %. In ultra‑low‑resource settings (0.5–1 hour of speech), DAMA delivers up to a 29 % relative WER reduction. Efficiency gains include a 24 % reduction in GPU memory usage and a 36 % speed‑up in training time. Crucially, the method preserves performance on high‑resource languages, avoiding the catastrophic forgetting observed with full fine‑tuning.

Overall, the study demonstrates that respecting the depth‑dependent representation hierarchy of multilingual speech models enables highly efficient, robust adaptation to low‑resource languages, offering a practical path toward scalable, multilingual ASR deployment.