GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning

Large language models (LLMs) exhibit pronounced position bias in long-context needle-in-haystack problems, systematically prioritizing the location of information over its relevance. While current mitigations rely on white-box access, this is effectively impossible for many state-of-the-art models. We introduce GOLD PANNING, a black-box Bayesian framework that performs inference-time active search over long contexts by (i) reordering documents to concentrate high-belief items in highly diagnostic positions (signal anchoring) and (ii) updating beliefs over document relevance from model outputs. Unlike conventional active learning, which prioritizes uncertainty reduction, GOLD PANNING leverages anchoring – once flagged, keep it in sight – to preserve weak cues. We implement this using iterative assignment derived from the model’s diagnosticity profile, which provably identifies a target among $N$ documents in $O(\log N)$ rounds, ensuring scalability to many-document settings.On needle-in-a-haystack retrieval and long-context QA, GOLD PANNING matches Permutation Self-Consistency’s target identification with $30–65%$ fewer queries and remains effective under calibration mismatch, suggesting coarse positional ordering drives performance gains. These results demonstrate that inherent model biases need not be failures, but can be used as tools for control.

💡 Research Summary

The paper tackles a fundamental limitation of large language models (LLMs) when reasoning over long contexts: a strong position bias that causes the model to favor information appearing at certain locations (typically the beginning or end) while neglecting content in the middle. Existing mitigations either require white‑box access to the model’s internals or rely on black‑box ensembling methods such as Permutation Self‑Consistency (PSC), which randomly reshuffle documents across many queries but treat each query independently and ignore information gathered from previous interactions.

GOLD PANNING (Strategic Context Shuffling for Needle‑in‑Haystack Reasoning) proposes a black‑box, inference‑time Bayesian active‑search framework that explicitly models this bias and exploits it. The method first calibrates a positional diagnosticity profile for a given LLM by estimating the true‑positive rate (TPR) and false‑positive rate (FPR) of each context slot using a small set of synthetic needle‑in‑haystack instances. The absolute difference |TPR − FPR|, i.e., Youden’s J statistic, quantifies how discriminative a position is; positions are then ranked by this diagnosticity.

During inference, the system maintains posterior beliefs bₜ,ᵢ = Pr(document i is relevant | all observations up to round t). At each round, documents are scored either by their current belief (GP‑BELIEF) or by their entropy (GP‑ENTROPY). Documents are sorted by score, positions are sorted by diagnosticity, and a greedy zip‑matching assigns the highest‑scoring document to the most diagnostic slot, the second‑highest to the second‑most diagnostic, and so on. The LLM is queried with this reordered prompt, and the model’s citations are interpreted as binary observations Oₜ,ᵢ. Using the calibrated TPR/FPR for the assigned slot, a Bayesian update (log‑odds addition) refines the beliefs.

Theoretical analysis shows that the log‑odds for a truly relevant document evolve as a random walk with expected increment μⱼ = KL(P₁ⱼ‖P₀ⱼ), the KL‑divergence between the observation distributions for relevance = 1 and relevance = 0 at position j. If the policy assigns, on average, a drift μ > 0 to the target, the number of rounds needed to isolate the target among N candidates scales as O((log N)/μ), i.e., logarithmic in the collection size. This yields an O(log N) query bound, dramatically better than the linear‑in‑T rounds required by PSC. Moreover, the information‑rate analysis proves that GP‑BELIEF’s greedy matching maximizes the expected drift compared with random placement, directly translating into fewer queries.

Empirically, the authors evaluate on synthetic needle‑in‑haystack tasks with up to 1024 documents and on multi‑document QA benchmarks (e.g., NaturalQuestions‑Open, TriviaQA‑Open). Calibration uses only a few hundred synthetic examples per model, and the resulting positional profiles transfer across model families (GPT‑3.5, GPT‑4, LLaMA‑2) and scales. GP‑BELIEF consistently identifies relevant documents with 30‑65 % fewer queries than PSC while achieving comparable or slightly higher F1 scores. GP‑ENTROPY, which pursues uncertainty reduction, underperforms GP‑BELIEF, confirming that anchoring high‑belief items in high‑diagnostic slots is more effective than probing uncertain items. The method remains robust under calibration mismatch, indicating that coarse positional ordering suffices.

In summary, GOLD PANNING demonstrates that LLM position bias need not be a flaw to be eliminated; instead, it can be harnessed as a controllable signal. By combining a simple Bayesian belief tracker with a greedy assignment based on calibrated diagnosticity, the framework achieves provable logarithmic query complexity and strong empirical gains, all while operating as a black‑box wrapper around any LLM API. The paper opens avenues for extending this approach to multi‑needle scenarios, dynamic document pools, and non‑textual modalities where positional effects also arise.