Towards Cross-lingual Values Assessment: A Consensus-Pluralism Perspective

While large language models (LLMs) have become pivotal to content safety, current evaluation paradigms primarily focus on detecting explicit harms (e.g., violence or hate speech), neglecting the subtler value dimensions conveyed in digital content. To bridge this gap, we introduce X-Value, a novel Cross-lingual Values Assessment Benchmark designed to evaluate LLMs’ ability to assess deep-level values of content from a global perspective. X-Value consists of more than 5,000 QA pairs across 18 languages, systematically organized into 7 core domains grounded in Schwartz’s Theory of Basic Human Values and categorized into easy and hard levels for discriminative evaluation. We further propose a unique two-stage annotation framework that first identifies whether an issue falls under global consensus (e.g., human rights) or pluralism (e.g., religion), and subsequently conducts a multi-party evaluation of the latent values embedded within the content. Systematic evaluations on X-Value reveal that current SOTA LLMs exhibit deficiencies in cross-lingual values assessment ($Acc < 77%$), with significant performance disparities across different languages ($ΔAcc > 20%$). This work highlights the urgent need to improve the nuanced, values-aware content assessment capability of LLMs. Our X-Value is available at: https://huggingface.co/datasets/Whitolf/X-Value.

💡 Research Summary

The paper introduces XValue, a cross‑lingual benchmark designed to evaluate large language models’ (LLMs) ability to assess the deep‑level values embedded in digital content. Recognizing that existing safety evaluations focus mainly on explicit harms such as violence, hate speech, or pornography, the authors argue that subtler, value‑related dimensions are largely ignored, allowing malicious actors to embed implicit bias or culturally sensitive stances without triggering conventional detectors.

XValue comprises more than 5,000 question‑answer (QA) pairs covering 18 languages that together represent over 75 % of the world’s population. The languages span diverse families (Indo‑European, Sino‑Tibetan, Afro‑Asiatic, Austronesian, etc.) and include English, Chinese, Hindi, Spanish, Arabic, French, Portuguese, Russian, Indonesian, Urdu, German, Japanese, Vietnamese, Korean, Thai, Turkish, Malay, and Italian. Each QA pair is constructed from real‑world online discussions (primarily Reddit) and is supplemented with two answer variants: a “normal” answer that reflects socially accepted values (sourced or generated by safety‑aligned LLMs) and a “risky” answer generated by an uncensored open‑source LLM to increase the presence of potentially inappropriate values.

The benchmark is organized around Schwartz’s Theory of Basic Human Values, which the authors adapt into seven high‑level domains relevant to content safety: Governance & Politics, Sovereignty & Security, History & Identity, Ethnicity & Equity, Belief & Expression, Gender & Rights, and Safety & Ethics. Within these domains, 16 specific values are mapped (e.g., power‑dominance, universalism‑tolerance, benevolence‑caring).

A novel two‑stage annotation pipeline is employed to capture cultural nuance. Stage 1 determines whether the issue addressed by a QA pair falls under “global consensus” (issues with broad international agreement such as human‑rights protection) or “pluralism” (issues where culturally divergent positions are legitimate, such as religious practices or political systems). Stage 2 then judges the answer’s alignment: for consensus issues the answer must explicitly uphold the consensus; for pluralistic issues the answer must remain neutral, inclusive, and avoid imposing a single value stance. Annotators—two native speakers per language, with a third adjudicator— assign binary labels (values‑appropriate vs. values‑inappropriate). The process prohibits the use of LLMs and allows only limited web searches, ensuring human‑centric judgment.

To stratify difficulty, the authors first run three strong LLMs (Qwen3‑Plus, GPT‑5.2, Gemini‑3‑Pro) on the entire pool of 54 000 generated QA pairs. If all three models agree on the value‑appropriateness label, the sample is marked “easy”; any disagreement marks it “hard”. This yields an easy‑subset that is relatively unambiguous and a hard‑subset that contains more subtle or culturally contingent value conflicts. For the final benchmark, roughly 300 samples per language are selected, balanced across domains and difficulty levels, resulting in a total of about 5 000 curated items.

Evaluation methodology defines accuracy (Acc) separately for easy, hard, and all subsets. Eight state‑of‑the‑art LLMs are tested: GPT‑5.2, Gemini‑3‑Flash‑Preview, Gemini‑3‑Pro‑Preview, Claude‑Opus‑4.5, Claude‑Opus‑4.6, Kimi‑k2.5‑thinking, Qwen3‑Plus, and Qwen3‑Max. Results show that while all models achieve >92 % accuracy on the easy subset, performance drops sharply on the hard subset, with the best hard‑subset accuracy below 66 %. Overall accuracy across the full benchmark hovers around 75 %, confirming a substantial gap between current capabilities and the demands of value‑aware safety assessment. Moreover, a pronounced language disparity is observed: the difference between the highest‑performing language and the lowest exceeds 20 %, indicating that non‑English models lag considerably.

Scaling experiments with the open‑source Qwen3 series (1.7 B to 235 B parameters) reveal a positive correlation between model size and values‑assessment accuracy, yet even the largest open‑source model fails to reach the easy‑subset performance of the top proprietary systems, underscoring that sheer scale is insufficient for mastering nuanced, culturally contingent value judgments.

The authors discuss several limitations. The “risky” answers are generated by proprietary, uncensored LLMs, which may affect reproducibility. The binary labeling scheme, while practical, inevitably compresses complex value nuances into a single decision. Some domains are under‑represented in low‑resource languages due to data scarcity, potentially biasing the evaluation.

In conclusion, XValue provides the first systematic, cross‑lingual benchmark for assessing LLMs’ capacity to recognize and respect deep human values in content. The empirical findings highlight that current SOTA models, despite impressive language understanding, are not yet reliable for value‑centric safety tasks, especially in multilingual and culturally diverse settings. The work calls for future research to incorporate value‑aware objectives during pre‑training, develop richer multi‑dimensional annotation schemas, and expand high‑quality data for under‑represented languages, thereby moving toward more ethically robust AI systems.