GeoRC: A Benchmark for Geolocation Reasoning Chains

Vision Language Models (VLMs) are good at recognizing the global location of a photograph – their geolocation prediction accuracy rivals the best human experts. But many VLMs are startlingly bad at explaining which image evidence led to their prediction, even when their location prediction is correct. The reasoning chains produced by VLMs frequently hallucinate scene attributes to support their location prediction (e.g. phantom writing, imagined infrastructure, misidentified flora). In this paper, we introduce the first benchmark for geolocation reasoning chains. We focus on the global location prediction task in the popular GeoGuessr game which draws from Google Street View spanning more than 100 countries. We collaborate with expert GeoGuessr players, including the reigning world champion, to produce 800 ground truth reasoning chains for 500 query scenes. These expert reasoning chains address hundreds of different discriminative visual attributes such as license plate shape, architecture, and soil properties to name just a few. We evaluate LLM-as-a-judge and VLM-as-a-judge strategies for scoring VLM-generated reasoning chains against our expert reasoning chains and find that Qwen 3 LLM-as-a-judge correlates best with human scoring. Our benchmark reveals that while large, closed-source VLMs such as Gemini and GPT 5 rival human experts at prediction locations, they still lag behind human experts when it comes to producing auditable reasoning chains. Open weights VLMs such as Llama and Qwen catastrophically fail on our benchmark – they perform only slightly better than a baseline in which an LLM hallucinates a reasoning chain with oracle knowledge of the photo location but no visual information at all. We believe the gap between human experts and VLMs on this task points to VLM limitations at extracting fine-grained visual attributes from high resolution images.

💡 Research Summary

GeoRC introduces the first benchmark that evaluates not only the geographic location prediction of images but also the quality of the reasoning chains that justify those predictions. The authors focus on the popular GeoGuessr game, which presents Google Street View panoramas from over 100 countries. They collaborated with three top‑ranked GeoGuessr players—including the reigning world champion—to produce 800 expert reasoning chains for 500 distinct scenes. Each chain lists discriminative visual attributes (e.g., license‑plate shape, architectural style, vegetation type, road markings) in a coarse‑to‑fine manner, ending with a final location guess and confidence level. The chains are annotated with up to three semantic categories (infrastructure, vegetation, architecture, etc.) to enable systematic analysis.

To assess how well Vision‑Language Models (VLMs) can generate comparable reasoning, the paper proposes three automated judging strategies. The first, “One‑to‑All LLM‑as‑a‑Judge,” asks a large language model (LLM) to score the similarity of every candidate statement against the full ground‑truth chain, producing precision, recall, and an F1 score. The second, “Key‑Points Guided LLM‑as‑a‑Judge,” compresses each statement into a few “key points,” embeds them with a sentence transformer, and uses cosine similarity with tuned thresholds to better align with human judgments. The third, “VLM‑as‑a‑Judge,” feeds the original image to a VLM, which reports how many candidate statements are visually corroborated, thereby detecting hallucinations directly from the visual modality.

Experiments were run on two Nvidia A40 GPUs using Qwen‑3‑4B‑Instruct as the LLM judge and Qwen‑2.5‑VL‑72B‑Instruct as the VLM judge. Closed‑source VLMs such as Gemini and GPT‑5 achieve near‑human accuracy on the pure location‑prediction task, but their reasoning‑chain F1 scores lag far behind human experts (≈0.42 vs. ≈0.78). Open‑source models (Llama‑2‑70B, Qwen‑2‑7B) perform only marginally better than a baseline that hallucinates a chain with oracle location knowledge but no visual grounding. Error analysis reveals four dominant failure modes: (i) omission of fine‑grained visual details, (ii) mis‑attribution of attributes (e.g., wrong road sign type), (iii) hallucination of non‑existent objects (phantom writing, imagined infrastructure), and (iv) irrelevant or axiomatic statements that do not aid localization.

Among the judges, Qwen 3 LLM‑as‑a‑Judge correlates best with human scoring (Pearson ≈ 0.71), indicating that LLMs excel at nuanced text‑to‑text similarity assessment. The VLM‑as‑a‑Judge, while conceptually attractive for detecting hallucinations, is limited by current VLMs’ inability to capture the minute visual cues required for high‑quality geolocation explanations.

The paper’s contributions are: (1) the GeoRC dataset—the first large‑scale collection of expert geolocation reasoning chains; (2) a human‑calibrated grading protocol based on precision, recall, and F1; (3) systematic evaluation of LLM‑as‑a‑judge and VLM‑as‑a‑judge methods; (4) quantitative benchmarking of VLM reasoning quality and a taxonomy of typical errors; (5) open‑sourcing of the dataset and the best LLM‑as‑a‑judge model for community use.

Overall, the study demonstrates a clear gap: while modern VLMs can match humans in predicting where a photo was taken, they fall short in articulating the visual evidence that supports that prediction. This gap points to fundamental limitations in extracting and reasoning over fine‑grained visual attributes from high‑resolution images. Future work should focus on improving VLMs’ high‑resolution feature extraction, integrating multimodal fine‑grained alignment objectives, and incorporating human‑like reasoning structures into prompting and model architecture to achieve both accurate location prediction and auditable, trustworthy explanations.