CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition

Modern deep learning models often achieve high overall performance, but consistently fail on specific subgroups. Group distributionally robust optimization (group DRO) addresses this problem by minimizing the worst-group loss, but it fails when group losses misrepresent performance differences between groups. This is common in domains like speech, where the widely used connectionist temporal classification (CTC) loss not only scales with input length but also varies with linguistic and acoustic properties, leading to spurious differences between group losses. We present CTC-DRO, which addresses the shortcomings of the group DRO objective by smoothing the group weight update to prevent overemphasis on consistently high-loss groups, while using input length-matched batching to mitigate CTC’s scaling issues. We evaluate CTC-DRO on the task of multilingual automatic speech recognition (ASR) across five language sets from the diverse ML-SUPERB 2.0 benchmark. CTC-DRO consistently outperforms group DRO and CTC-based baseline models, reducing the worst-language error by up to 47.1% and the average error by up to 32.9%. CTC-DRO can be applied to ASR with minimal computational costs, and, while motivated by multilingual ASR, offers the potential for reducing group disparities in other domains with similar challenges.

💡 Research Summary

The paper tackles a pervasive problem in modern deep‑learning systems: while overall accuracy may be high, performance on certain sub‑populations can be dramatically worse. In multilingual automatic speech recognition (ASR) this manifests as large language‑wise error gaps caused by differences in data volume, utterance length, speaker characteristics, and recording conditions. Group distributionally robust optimization (group DRO) is a popular remedy that up‑weights high‑loss groups during training, aiming to minimize the worst‑group loss. However, the authors identify a critical failure mode when the loss function itself is not comparable across groups.

Connectionist Temporal Classification (CTC), the de‑facto loss for end‑to‑end ASR, scales with input length D and output length U. Longer utterances produce higher raw CTC values even if the transcription is relatively accurate. Consequently, groups (languages) with longer average utterances (e.g., Spanish) receive disproportionately large loss values, prompting the standard group‑DRO algorithm to assign them excessive weight. This “over‑emphasis” harms other languages, leading to sub‑optimal overall performance.

CTC‑DRO is introduced to resolve both the scaling bias and the over‑weighting issue. It comprises two complementary mechanisms:

1. Length‑matched batching – During each training step a single language g is sampled, and utterances from that language are added to a batch until the total audio duration approximates a fixed target d. By ensuring that each group’s batch contributes roughly the same total duration, the summed CTC loss becomes comparable across groups, effectively normalizing away the length‑dependent scaling.

2. Smoothed group‑weight update – Instead of the classic Hedge update q_g ← q_g·exp(η_q·L_g)/∑_h q_h·exp(η_q·L_h), the authors propose
   q_g ← q_g·exp(η_q·L_g/(q_g+α)) / ∑_h q_h·exp(η_q·L_h/(q_h+α)),
   where α≥0 is a smoothing hyper‑parameter. When α→0 the update becomes highly sensitive to the current weight, preventing any single group from dominating; when α→∞ the rule collapses to the original group‑DRO update. Theoretical analysis shows that the optimal solution satisfies q_g+α ∝ L_g·P_h L_h, confirming that weights still increase with loss but in a controlled, smoother fashion.

Algorithm 1 details the full training loop: sample a group, build a length‑matched batch, compute per‑utterance CTC losses, accumulate them per group, update q after each group has been seen, and finally perform a weighted gradient step on the model parameters. The method requires only a scalar weight per group, preserving the low computational overhead of standard group DRO.

Experimental setup – The authors fine‑tune two large self‑supervised multilingual encoders (XLS‑R and MMS) on the ML‑SUPERB 2.0 benchmark, adding two transformer layers and a CTC head that predicts a language token followed by characters. Five language sets are randomly drawn from the benchmark, each exhibiting distinct utterance‑length distributions and data‑size imbalances. Baselines include (i) vanilla CTC fine‑tuning and (ii) standard group DRO applied to the same models.

Results – Across all five language sets, CTC‑DRO consistently reduces the worst‑language word error rate (WER) by up to 47.1 % relative and improves average WER by up to 32.9 % relative compared to the vanilla CTC baseline. Against standard group DRO, CTC‑DRO yields an average 22 % relative reduction in worst‑language WER and a 15 % reduction in mean WER. Notably, languages with longer utterances no longer dominate the loss landscape; the length‑matched batching neutralizes the scaling effect, while the smoothed weight update prevents the algorithm from over‑compensating for any single high‑loss group. Training curves show smoother convergence and more balanced weight trajectories.

Efficiency and broader impact – Because only per‑group scalar weights are stored and the extra computation is limited to batch sampling and a logarithmic weight update, CTC‑DRO adds negligible overhead to existing pipelines. The authors argue that the same principles apply to any domain where loss magnitudes differ across groups (e.g., medical imaging, legal document classification), suggesting a general recipe for robust, fair training when group‑wise loss comparability cannot be guaranteed.

In summary, CTC‑DRO offers a practical, theoretically grounded augmentation to group DRO that specifically addresses the idiosyncrasies of CTC‑based speech models. By normalizing loss scales through length‑matched batches and tempering group‑weight dynamics with a smoothing hyper‑parameter, it achieves substantial reductions in language‑wise performance disparities while maintaining computational efficiency, making it a compelling tool for building more equitable multilingual ASR systems.