Benchmarking Automatic Speech Recognition Models for African Languages

Automatic speech recognition (ASR) for African languages remains constrained by limited labeled data and the lack of systematic guidance on model selection, data scaling, and decoding strategies. Large pre-trained systems such as Whisper, XLS-R, MMS, and W2v-BERT have expanded access to ASR technology, but their comparative behavior in African low-resource contexts has not been studied in a unified and systematic way. In this work, we benchmark four state-of-the-art ASR models across 13 African languages, fine-tuning them on progressively larger subsets of transcribed data ranging from 1 to 400 hours. Beyond reporting error rates, we provide new insights into why models behave differently under varying conditions. We show that MMS and W2v-BERT are more data efficient in very low-resource regimes, XLS-R scales more effectively as additional data becomes available, and Whisper demonstrates advantages in mid-resource conditions. We also analyze where external language model decoding yields improvements and identify cases where it plateaus or introduces additional errors, depending on the alignment between acoustic and text resources. By highlighting the interaction between pre-training coverage, model architecture, dataset domain, and resource availability, this study offers practical and insights into the design of ASR systems for underrepresented languages.

💡 Research Summary

This paper presents a systematic benchmark of four state‑of‑the‑art pre‑trained automatic speech recognition (ASR) models—Whisper, XLS‑R, Massively Multilingual Speech (MMS), and W2v‑BERT—across thirteen African languages. The authors fine‑tune each model on progressively larger subsets of transcribed speech, ranging from 1 hour up to 400 hours (the latter only for Swahili, which has the most data). In addition to reporting word error rates (WER), they investigate how model architecture, pre‑training coverage, data scaling, and external language‑model (LM) decoding interact.

Data and Experimental Setup
The speech corpora are drawn from publicly available sources such as Google FLEURS, Common Voice, NCHLT, AfriVox, and ALFF, covering a variety of speaking styles (read, conversational, descriptive). For each language, a fixed test set is held constant across all training sizes to ensure fair comparison. Text corpora for twelve of the languages are compiled from news, health, education, agriculture, and religious domains; 5‑gram KenLM models with modified Kneser‑Ney smoothing are trained for use with XLS‑R and W2v‑BERT. Kinyarwanda is excluded from XLS‑R and W2v‑BERT fine‑tuning because it appears extensively in their pre‑training data, avoiding leakage.

All models are fine‑tuned with BF16 precision, AdamW optimizer, linear learning‑rate schedule with a 10 % warm‑up, and early stopping based on validation loss. Feature extractors are frozen for XLS‑R, MMS, and W2v‑BERT, while Whisper is fine‑tuned end‑to‑end. Training is performed on NVIDIA A40, L4, RTX A6000, and A100 GPUs, with batch size 64 and gradient accumulation to fit memory constraints.

Key Findings

1. Data‑Efficiency in Extreme Low‑Resource Settings

   • With only 1–10 hours of labeled speech, MMS and W2v‑BERT achieve the lowest WERs, outperforming Whisper and XLS‑R by roughly 12–15 percentage points on average. Their massive self‑supervised pre‑training (≈500 k hours for MMS, 4.5 M hours for W2v‑BERT) provides robust acoustic representations that require minimal supervision.

2. Scaling Behaviour

   • XLS‑R shows the steepest performance gains between 20 and 50 hours. For example, Afrikaans drops from 38.6 % to 2.8 % WER and Xhosa from 54.7 % to 8.5 % within this range, indicating strong learning from modest data when the language is already represented in the pre‑training corpus.
   • Beyond 100 hours, the benefit of additional data becomes language‑dependent. Swahili continues to improve up to 400 hours (26.4 % → 11.4 % WER), whereas Luganda plateaus after 100 hours (≈42 % → 40 %). This suggests that linguistic complexity, orthographic depth, and overlap with pre‑training data modulate the marginal returns of scaling.

3. Mid‑Resource Sweet Spot for Whisper

   • Whisper‑small, trained on 680 k hours of labeled speech and translation in 97 languages, excels in the 50–200 hour regime, delivering the lowest WERs for several languages and showing robustness to domain and noise variations. Its encoder‑decoder transformer benefits from multitask pre‑training, which appears to pay off once a moderate amount of fine‑tuning data is available.

4. Impact of External Language‑Model Decoding

   • Adding a 5‑gram LM improves XLS‑R and W2v‑BERT in the ≤20 hour regime by an average of 4.2 percentage points, especially for agglutinative or morphologically rich languages (e.g., Lingala, Shona). However, as the acoustic model becomes stronger with more data, LM gains diminish and can even cause degradation, likely because the LM imposes constraints that conflict with the model’s learned lexical probabilities.

5. Pre‑Training Corpus Overlap Matters

   • For Kinyarwanda, which is heavily represented in the pre‑training data of XLS‑R and W2v‑BERT, the authors deliberately omit fine‑tuning with those models to avoid inflated results. Whisper, MMS, and wav2vec2‑large are evaluated instead, and Whisper emerges as the most reliable choice, underscoring that the degree of overlap between target language data and pre‑training corpora is a decisive factor in model selection.

Practical Recommendations

• Extreme low‑resource (≤10 h): Prefer MMS or W2v‑BERT, which are most data‑efficient.
• Low‑to‑mid resource (20–100 h): XLS‑R offers the best scaling, especially when the language appears in its pre‑training set.
• Mid‑to‑high resource (>100 h): Whisper becomes competitive, particularly for noisy or domain‑shifted data.
• External LM usage: Deploy n‑gram LMs only when labeled data is scarce and the language exhibits high morphological complexity; disable LM decoding once sufficient acoustic data is available.
• Check pre‑training coverage: Verify whether the target language is already present in a model’s pre‑training corpus; if so, consider zero‑shot evaluation or exclude that model to prevent leakage.

Contribution to the Field
By evaluating all four models under identical conditions, the paper fills a critical gap in African ASR research, which has traditionally reported isolated results on single models or datasets. The authors provide a nuanced, data‑driven framework that links pre‑training breadth, model architecture, data scale, and decoding strategy to performance outcomes. This framework equips researchers and practitioners with actionable guidance for building ASR systems in under‑represented languages, helping to prioritize data collection efforts, model selection, and decoding pipelines based on concrete empirical evidence.