Towards Reliable Machine Translation: Scaling LLMs for Critical Error Detection and Safety

Machine Translation (MT) plays a pivotal role in cross-lingual information access, public policy communication, and equitable knowledge dissemination. However, critical meaning errors, such as factual distortions, intent reversals, or biased translations, can undermine the reliability, fairness, and safety of multilingual systems. In this work, we explore the capacity of instruction-tuned Large Language Models (LLMs) to detect such critical errors, evaluating models across a range of parameters using the publicly accessible data sets. Our findings show that model scaling and adaptation strategies (zero-shot, few-shot, fine-tuning) yield consistent improvements, outperforming encoder-only baselines like XLM-R and ModernBERT. We argue that improving critical error detection in MT contributes to safer, more trustworthy, and socially accountable information systems by reducing the risk of disinformation, miscommunication, and linguistic harm, especially in high-stakes or underrepresented contexts. This work positions error detection not merely as a technical challenge, but as a necessary safeguard in the pursuit of just and responsible multilingual AI. The code will be made available at GitHub.

💡 Research Summary

The paper addresses a critical gap in machine translation (MT) evaluation: the inability of conventional metrics such as BLEU, METEOR, and COMET to capture meaning‑critical errors that can distort facts, reverse intent, or embed bias. To fill this gap, the authors define a new binary classification task called Critical Error Detection (CED). Given a source sentence (English) and its translation (German), a model must label the translation as either “ERR” (contains a meaning‑critical error) or “NOT” (faithful translation).

Three publicly available English‑German datasets are used for systematic evaluation: WMT‑21, WMT‑22, and the synthetic‑plus‑human‑validated SynCED‑EnDe 2025. WMT‑22 provides a large, imbalanced corpus (≈155 k training pairs) reflecting real‑world error frequencies, while SynCED offers a balanced set of 8 k training pairs with controlled perturbations. All datasets include binary error annotations, enabling consistent cross‑benchmark comparison.

The study evaluates two broad families of models. Encoder‑only baselines (BERT‑base, ModernBERT‑base/large, mmBERT, XLM‑R‑large) serve as strong multilingual representation learners but lack generative reasoning. Decoder‑based large language models (LLMs) include GPT‑4o‑mini, GPT‑4o, LLaMA‑3.1‑8B‑Instruct, LLaMA‑3.3‑70B‑Instruct, and GPT‑OSS‑20B/120B. These LLMs are instruction‑tuned, giving them strong factual alignment and chain‑of‑thought capabilities.

Four adaptation regimes are explored for each LLM:

1. Zero‑shot (P0) – a concise instruction prompts the model to output a single token (“ERR” or “NOT”).
2. Few‑shot (P1) – the same instruction is preceded by eight labeled examples (5 ERR, 3 NOT) to provide in‑context learning.
3. Prompt tuning (P2‑P4) – model‑specific templates are crafted to clarify task boundaries, especially benefiting smaller models.
4. Fine‑tuning – parameter‑efficient LoRA adapters are trained on the merged 170 k training pairs; after fine‑tuning, models are evaluated with the original zero‑shot prompt to isolate the effect of weight updates.

Performance is measured primarily with Matthews Correlation Coefficient (MCC), which is robust to class imbalance, complemented by class‑wise F1 scores for ERR and NOT.

Key empirical findings:

• Encoder‑only baselines achieve respectable MCC scores on the larger, clearer datasets (up to 0.88 on WMT‑22 and SynCED) but struggle on WMT‑21 where subtle meaning shifts dominate, confirming that static sentence embeddings alone cannot reliably detect nuanced errors.
• Decoder LLMs show competitive zero‑shot performance, with GPT‑4o and LLaMA‑3.3‑70B reaching MCC ≈ 0.33–0.62 depending on the dataset. Few‑shot prompting consistently improves ERR recall, raising MCC by 0.05–0.10 and especially boosting the minority class F1.
• Prompt‑tuned variants (P2‑P4) yield the largest gains for smaller models (e.g., GPT‑4o‑mini improves from MCC 0.30 to ≈ 0.45). For already well‑aligned large models, over‑specification can cause slight drops, highlighting a trade‑off between model size and prompt complexity.
• Majority‑voting ensembles (three stochastic generations, temperature 0.2) reduce variance and correct isolated misclassifications, further increasing MCC across the board.
• Fine‑tuned LLMs outperform all zero‑shot and few‑shot configurations, with MCC gains up to 0.70 for GPT‑4o‑mini and 0.65 for LLaMA‑3.1‑8B, demonstrating that modest parameter updates on task‑specific data substantially enhance critical error detection.

Societal and ethical considerations: The authors argue that CED should be treated as a safety layer rather than a mere technical metric. In high‑stakes domains such as healthcare, legal aid, and finance, undetected translation errors can propagate misinformation, exacerbate inequities, and erode public trust. They propose a human‑in‑the‑loop workflow where the LLM’s error score triggers human review, thereby mitigating automation bias and false‑positive risks. The paper also acknowledges limitations: the study is confined to English‑German, error labeling may contain subjectivity, and the cost of large‑scale LLM inference may hinder deployment in resource‑constrained settings.

Future directions: Extending CED to a broader set of languages, incorporating richer error taxonomies, improving label quality, and developing cost‑effective, distilled models are identified as essential next steps.

In summary, the paper provides a thorough scaling study of instruction‑tuned LLMs for critical error detection in machine translation. It demonstrates that model size, instruction alignment, and modest fine‑tuning jointly drive substantial performance gains over traditional encoder‑only baselines. By framing error detection as a safeguard for trustworthy multilingual communication, the work bridges technical advancement with broader societal responsibility, offering a concrete pathway toward safer, more inclusive AI‑mediated information access.