Approaching the linguistic complexity

We analyze the rank-frequency distributions of words in selected English and Polish texts. We compare scaling properties of these distributions in both languages. We also study a few small corpora of Polish literary texts and find that for a corpus consisting of texts written by different authors the basic scaling regime is broken more strongly than in the case of comparable corpus consisting of texts written by the same author. Similarly, for a corpus consisting of texts translated into Polish from other languages the scaling regime is broken more strongly than for a comparable corpus of native Polish texts. Moreover, based on the British National Corpus, we consider the rank-frequency distributions of the grammatically basic forms of words (lemmas) tagged with their proper part of speech. We find that these distributions do not scale if each part of speech is analyzed separately. The only part of speech that independently develops a trace of scaling is verbs.

💡 Research Summary

The paper investigates linguistic complexity through the lens of Zipf’s law, which predicts that the frequency of a word is inversely proportional to its rank in a sufficiently large text. Using a combination of classic literary examples and modern corpora, the authors compare English and Polish texts, examine the influence of authorship, translation, and part‑of‑speech (POS) on the rank‑frequency scaling, and discuss what these findings imply for quantitative linguistics.

First, the authors replicate Zipf’s original analysis on James Joyce’s Ulysses in English and its Polish translation. Both texts display power‑law behavior over three decades (ranks 10–10 000), but the scaling exponent α differs: α≈1.05 for the English original versus α≈0.90 for the Polish version. The lower exponent in Polish is interpreted as a consequence of the language’s richer inflectional morphology, which requires a larger set of word forms to convey the same narrative content.

Second, the study turns to the British National Corpus (BNC). After lemmatizing the corpus, the authors extract the most frequent nouns, verbs, adjectives, and adverbs and plot their individual rank‑frequency curves. While the whole corpus follows Zipf’s law (α≈1) up to several thousand ranks, the POS‑specific curves largely lose this scaling. Only verbs retain a faint power‑law trace (α≈1), suggesting that the action‑oriented class of words maintains a more uniform distribution across frequencies than other lexical categories.

Third, the authors construct two Polish corpora of comparable size (~1.3 million tokens). Corpus A contains 26 novels and stories by a single author (Andrzej Sapkowski); Corpus B aggregates 41 works by eight different authors. Both exhibit Zipf‑like scaling for low to moderate ranks, but beyond ranks 6 000–8 000 Corpus B decays faster, indicating a breakdown of the single‑exponent regime. The authors argue that the continuous narrative of a single author creates long‑range correlations that sustain scaling, whereas mixing different authorial styles disrupts these correlations.

Fourth, a comparison is made between native Polish texts (45 works) and Polish translations of foreign literature (30 works), each corpus comprising about 2.3 million tokens. For ranks below 2 000 both corpora share a similar exponent (α≈0.94). At higher ranks the translated corpus shows markedly lower frequencies, i.e., a stronger deviation from Zipf’s law. The authors attribute this to the translator’s constraints: preserving meaning and style limits lexical diversity, especially for low‑frequency words, while the original author enjoys full lexical freedom.

Throughout, the methodology involves standard preprocessing (tokenization, lemmatization, POS‑tagging), log‑log plotting of rank versus frequency, and linear regression to estimate α. The paper acknowledges potential biases: the selection of “Ulysses” as a test case, differences in corpus composition, and the lack of a deeper morphological analysis that could more precisely quantify the impact of inflectional richness.

The discussion highlights two main implications. First, Zipf’s law remains robust at the macro level (the whole language system) but is fragile when examined within sub‑systems such as POS classes, authorial ensembles, or translated material. The persistence of scaling for verbs may reflect a deeper connection between the principle of least effort and action‑oriented lexical items. Second, the traditional view that larger corpora are always superior for statistical linguistic analysis is challenged; aggregating heterogeneous texts can erase valuable information about long‑range dependencies present in single‑author works.

In conclusion, the study confirms that Zipf’s law is a useful indicator of linguistic complexity but must be interpreted with caution when dealing with finer linguistic structures. Future work could extend the analysis to other language families, incorporate morpheme‑level statistics, and explore how modern neural language models capture or violate the observed scaling phenomena.