Geography According to ChatGPT -- How Generative AI Represents and Reasons about Geography

Understanding how AI will represent and reason about geography should be a key concern for all of us, as the broader public increasingly interacts with spaces and places through these systems. Similarly, in line with the nature of foundation models, our own research often relies on pre-trained models. Hence, understanding what world AI systems construct is as important as evaluating their accuracy, including factual recall. To motivate the need for such studies, we provide three illustrative vignettes, i.e., exploratory probes, in the hope that they will spark lively discussions and follow-up work: (1) Do models form strong defaults, and how brittle are model outputs to minute syntactic variations? (2) Can distributional shifts resurface from the composition of individually benign tasks, e.g., when using AI systems to create personas? (3) Do we overlook deeper questions of understanding when solely focusing on the ability of systems to recall facts such as geographic principles?

💡 Research Summary

The paper investigates how generative AI systems such as ChatGPT represent and reason about geography, arguing that accuracy alone is insufficient to evaluate their impact. It frames “representation” both as an epistemological construct and as a matter of geographic coverage, noting that models can over‑represent some regions while under‑representing others, leading to biased outputs. Three exploratory vignettes illustrate distinct challenges. Vignette I examines “default strength” and brittleness: across 200 queries, GPT‑5.1 repeatedly returns Japan for “Name a country, please.” but switches to Canada for the semantically identical prompt “Please name a country.” The authors formalize default strength as the minimum temperature needed for non‑default answers to appear, showing that newer model generations tend to exhibit stronger defaults. Vignette II shows how seemingly benign tasks can combine to produce hidden distribution shifts. Using GPT‑4o, the authors generated 50 synthetic Los Angeles personas multiple times, finding age and occupation skews compared with census data. When the same synthetic population was later asked to assign criminal records, the racial distribution of alleged offenders diverged markedly from real arrest statistics, highlighting the difficulty of evaluating bias in composite workflows even when safety filters are in place. Vignette III distinguishes between factual recall and genuine understanding. While models correctly describe Zipf’s law for city sizes, they fail to apply it when asked to construct a plausible set of 30 cities for a fictional nation, often violating the total population constraint. This gap underscores that models can “know” a principle without being able to use it autonomously. The authors conclude that studying how AI constructs geographic knowledge—its defaults, sensitivity to phrasing, compositional bias, and ability to apply theory—is essential for responsible deployment in tourism, urban planning, real‑estate, and other geospatial domains. Future work should develop quantitative metrics for default strength, robust debiasing strategies that respect cultural norms, and evaluation frameworks that capture both representation and reasoning capabilities.