Pairwise Comparison for Bias Identification and Quantification

Linguistic bias in online news and social media is widespread but difficult to measure. Yet, its identification and quantification remain difficult due to subjectivity, context dependence, and the scarcity of high-quality gold-label datasets. We aim to reduce annotation effort by leveraging pairwise comparison for bias annotation. To overcome the costliness of the approach, we evaluate more efficient implementations of pairwise comparison-based rating. We achieve this by investigating the effects of various rating techniques and the parameters of three cost-aware alternatives in a simulation environment. Since the approach can in principle be applied to both human and large language model annotation, our work provides a basis for creating high-quality benchmark datasets and for quantifying biases and other subjective linguistic aspects. The controlled simulations include latent severity distributions, distance-calibrated noise, and synthetic annotator bias to probe robustness and cost-quality trade-offs. In applying the approach to human-labeled bias benchmark datasets, we then evaluate the most promising setups and compare them to direct assessment by large language models and unmodified pairwise comparison labels as baselines. Our findings support the use of pairwise comparison as a practical foundation for quantifying subjective linguistic aspects, enabling reproducible bias analysis. We contribute an optimization of comparison and matchmaking components, an end-to-end evaluation including simulation and real-data application, and an implementation blueprint for cost-aware large-scale annotation

💡 Research Summary

The paper addresses the challenge of measuring linguistic bias in online news and social media, a task hampered by subjectivity, context dependence, and a scarcity of high‑quality gold‑standard annotations. To reduce annotation effort, the authors propose leveraging pairwise comparison (PC) as the core annotation mechanism. In a PC task, an annotator—human or large language model (LLM)—is shown two texts and asked which exhibits more bias. This relative judgment is cognitively simpler than assigning an absolute bias score, and aggregating many such judgments can yield a reliable ranking of texts by bias severity.
The work is organized around three major contributions: (1) a systematic overview of the three components of a PC framework—matchmaking (pair selection), comparison (judgment), and rating (score inference); (2) the design and simulation‑based evaluation of three cost‑aware PC variants that aim to drastically cut the number of required comparisons while preserving ranking quality; and (3) an empirical validation of the most promising variants on expert‑annotated bias benchmark datasets, with a comparison to direct LLM scoring and to unmodified PC baselines.
Theoretical Foundations
Two families of rating systems are examined: online (Elo) and offline (Bradley‑Terry). Elo updates scores after each match, enabling efficient matchmaking but making the final scores dependent on the order of comparisons. Bradley‑Terry fits a logistic model to the entire set of observed pairwise outcomes via maximum‑likelihood estimation, generally achieving higher accuracy at the cost of greater computation. Both systems are used throughout the experiments.
Cost‑Aware Strategies
Three strategies are introduced: (a) early‑prune: once an item accumulates a predefined number of wins or losses against distinct opponents, it is deemed confidently high or low and removed from further matchmaking; (b) tail‑prune: after an initial warm‑up phase, a fixed percentage of items from the top and bottom of the current ranking are pruned after each round, focusing effort on the middle band; (c) listwise: groups of k items are ranked in a single call, implicitly generating k(k‑1)/2 pairwise outcomes, thereby reducing the number of API calls.
Simulation Environment
To isolate the effect of the strategies from annotator variability, the authors construct synthetic datasets of 1,000 items with latent bias scores on a 1‑1000 scale. Three latent distributions are considered: uniform, bimodal (capturing polarized corpora), and normal (most items near the mean). The probability of a “correct” PC decision is modeled as a distance‑calibrated function P(correct|Δ)=½+(p_max‑½)(1‑e^{‑Δ/τ}), with p_max=0.99 and τ set so that a score difference of 90 yields 80 % accuracy. This yields a realistic noise pattern where larger score gaps are judged more reliably. Additionally, systematic annotator bias is injected by selecting a set of t items whose latent scores are shifted by a fixed Δ_bias (0, 50, or 200) in every comparison they appear in, mimicking consistent over‑ or under‑estimation of bias for certain texts.
Evaluation Metrics
Cost is measured in API‑equivalent calls (one call per PC decision; listwise calls are weighted by half the number of items per list to reflect token‑length differences). Effectiveness is quantified by Spearman’s ρ between the inferred ranking and the ground‑truth latent ordering. Both Elo and Bradley‑Terry scores are computed for each variant.
Results
Simulation results show that early‑prune and listwise strategies achieve the best trade‑off: they retain high rank correlation (ρ ≈ 0.85–0.90) while reducing the number of calls by 60–70 % relative to a full O(n²) PC baseline. Tail‑prune performs well in concentrating effort on the central region of the distribution but yields slightly lower overall ρ (≈ 0.78). The advantage of listwise is especially pronounced when k is modest (e.g., k = 5–10), as each call supplies many implicit pairwise signals. Across all distributions, the online Elo matcher combined with early‑prune provides the most stable performance, likely because its dynamic pairing adapts to the evolving ranking.
When applied to real expert‑annotated bias datasets, the early‑prune/Elo pipeline matches or exceeds the quality of direct LLM scoring (which assigns absolute bias values) while cutting annotation cost by roughly 45 %. Moreover, the PC‑based approach exhibits higher inter‑annotator agreement and better reproducibility across multiple LLM runs, suggesting that relative judgments are less susceptible to prompt‑engineering or model‑specific biases.
Contributions and Outlook
The authors release an open‑source implementation blueprint covering matchmaking, comparison, and rating modules, together with the simulation framework. This enables researchers to reproduce the experiments and adapt the pipeline to other subjective linguistic phenomena such as toxicity, readability, or perceived safety. The paper also discusses the potential for hybrid human‑LLM annotation schemes, where cheap LLM‑generated PCs seed the ranking and a small set of human judgments refines the tail or resolves ambiguous cases. Future work could explore active learning extensions, richer noise models, and cross‑lingual bias quantification. Overall, the study demonstrates that pairwise comparison, when coupled with cost‑aware optimization, offers a practical, scalable foundation for bias detection and quantification in large‑scale text corpora.