Evaluating Extremely Low-Resource Machine Translation: A Comparative Study of ChrF++ and BLEU Metrics

Evaluating machine translation (MT) quality in extremely low-resource language (ELRL) scenarios poses unique challenges, as widely used metrics such as BLEU, effective in high-resource settings, often misrepresent quality in data-scarce contexts. This work presents a comparative analysis of BLEU, an n-gram-based metric, and ChrF++, a character-based metric, for MT evaluation in ELRL settings. We examine how each metric responds to translation artifacts, including hallucinations, repetition, source-text copying, and diacritic (\textit{matra}) variations across three ELRLs: Magahi, Bhojpuri, and Chhattisgarhi, with a focus on outputs from large language models (LLMs) and neural MT (NMT) systems. While recent work often relies solely on ChrF++, our findings show that BLEU, despite its lower absolute scores, provides complementary lexical-precision insights that improve interpretability.

💡 Research Summary

This paper investigates the challenges of automatic evaluation for machine translation (MT) in extremely low‑resource language (ELRL) settings, focusing on three Indo‑Aryan languages—Magahi, Bhojpuri, and Chhattisgarhi—that suffer from severe data scarcity. The authors compare two widely used automatic metrics: BLEU, an n‑gram precision‑based score, and ChrF++, a character‑level F‑score that is more tolerant of morphological variation. They argue that while BLEU has become the de‑facto standard in high‑resource contexts, its reliance on exact word‑order and lexical overlap can severely penalize translations in ELRLs, especially those with rich inflectional morphology and script‑specific diacritics (matras). ChrF++ mitigates some of these issues by operating on character n‑grams, but it can over‑estimate quality when surface similarity masks deeper semantic errors such as hallucinations or source‑copying.

The experimental setup includes three translation systems: Aya‑101 (a multilingual model covering 101 languages), Airavata (an Indic‑focused LLM), and mT5‑Large (a conventional neural MT model). All models are fine‑tuned on the 6,192‑sentence NLLB Seed corpus and evaluated on the 1,012‑sentence FLORES‑200 development set. The authors test both English → target and Hindi → target directions, as well as the reverse, and compute BLEU and ChrF++ using SacreBLEU with standard tokenization.

Results (Tables 1‑3) reveal large divergences between the two metrics. In many cases ChrF++ scores are high (often above 40) while BLEU remains low (single‑digit values). A striking example is Hindi → Magahi translation where the model largely copies the source sentence: ChrF++ reaches 41.43 but BLEU drops to 18.09, indicating that character overlap is preserved but lexical‑level precision is poor. Conversely, modest increases in ChrF++ sometimes accompany dramatic BLEU improvements, reflecting genuine quality gains such as correct diacritic placement and better n‑gram alignment.

The authors conduct a fine‑grained error analysis and identify six recurring BLEU–ChrF++ divergence patterns: (1) simultaneous drops (poor overall quality), (2) stable ChrF++ with sharp BLEU decline (hallucinations), (3) ChrF++ rise with BLEU fall (source copying), (4) ChrF++ rise with BLEU slight fall (surface similarity despite semantic drift), (5) minor ChrF++ rise with major BLEU increase (accurate morphology and fluency), and (6) ChrF++ decrease with BLEU rise (lexical precision outweighing character similarity). Representative translation examples illustrate each pattern, confirming that BLEU is sensitive to n‑gram precision, brevity penalties, and diacritic errors, while ChrF++ captures surface character overlap but can miss deeper adequacy problems.

A supplemental experiment reduces the training data by 20 %. The same divergence trends persist, underscoring that data scarcity amplifies metric discrepancies. The authors also discuss learned metrics such as COMET and BLEURT, noting that their reliance on high‑resource pretraining makes them unreliable for ELRLs, especially when they inadvertently map ELRL outputs to related high‑resource languages (e.g., treating Magahi as Hindi).

In conclusion, the paper argues that BLEU and ChrF++ provide complementary signals in ELRL MT evaluation. BLEU offers lexical‑precision and length‑sensitivity cues that can flag hallucinations, repetitions, and diacritic errors, while ChrF++ supplies a robust measure of character‑level overlap that tolerates morphological variation. Practitioners are advised to examine both scores jointly; large divergences between them can serve as diagnostic indicators of specific translation faults. The study fills a gap in the literature by focusing exclusively on ELRL contexts, providing linguistically grounded insights, and offering concrete guidelines for metric interpretation when evaluating LLM‑generated and conventional NMT outputs in data‑scarce languages.