A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments

This paper presents an embedding-based approach to detecting variation without relying on prior normalisation or predefined variant lists. The method trains subword embeddings on raw text and groups related forms through combined cosine and n-gram similarity. This allows spelling and morphological diversity to be examined and analysed as linguistic structure rather than treated as noise. Using a large corpus of Luxembourgish user comments, the approach uncovers extensive lexical and orthographic variation that aligns with patterns described in dialectal and sociolinguistic research. The induced families capture systematic correspondences and highlight areas of regional and stylistic differentiation. The procedure does not strictly require manual annotation, but does produce transparent clusters that support both quantitative and qualitative analysis. The results demonstrate that distributional modelling can reveal meaningful patterns of variation even in ‘’noisy’’ or low-resource settings, offering a reproducible methodological framework for studying language variety in multilingual and small-language contexts.

💡 Research Summary

This paper introduces a novel, reproducible framework for detecting lexical and orthographic variation directly from raw text without relying on pre‑existing variant lists or normalization pipelines. The authors train FastText subword embeddings on a large corpus of Luxembourgish user comments (approximately 1.42 million comments spanning 2008–2024). By representing words as vectors built from fixed character n‑grams (3–7 grams), the model captures both semantic context and fine‑grained orthographic detail, even for low‑frequency forms.

After training, the method constructs a candidate lexicon of tokens meeting a minimum frequency threshold (≥10). For each token, the top‑N nearest neighbours are retrieved based on cosine similarity (≥0.73) and a character‑n‑gram Jaccard overlap (≥0.73). The two similarity scores are combined into a cohesion score (harmonic mean). Two clustering modes are offered: an “open” star‑shaped neighbourhood and a “strict” graph‑based approach. The authors employ the strict mode, building an undirected graph where edges exist if both similarity criteria are satisfied, then extracting connected components as variant families. Additional pruning removes families that are too small, overly frequent, or fail to meet a minimum user‑coverage requirement (≥3 distinct users).

The resulting families are exported as JSONL (full details) and CSV (summary statistics). The authors then perform a manual, bottom‑up inspection, assigning each family to one or more of seven categories: orthographic, morphological, lexical, collocation, tokenisation, regional, and other. Orthographic families include spellings that violate the official orthography (e.g., “laang” vs. “lang”) or reflect phonological spelling choices (e.g., “krng”, “srng”, “dng”, “êng”, “öng”). Morphological families capture inflectional variants such as verb conjugations (“fëllen”, “fëllt”, “fëlle”) and noun‑adjective declensions. Lexical families group semantically related synonyms or antonyms (“méi” vs. “manner”, “dass” vs. “datt”). Collocation families consist of words that frequently co‑occur as a fixed phrase (e.g., “Gott säi Dank”). Tokenisation families differentiate forms with and without the definite article “d’”. Regional families isolate variants that appear predominantly among users from specific geographic areas. The “other” category gathers families that do not fit the previous types.

Quantitatively, the authors identify 394 orthographic, 222 morphological, 115 lexical, 21 collocation, 14 tokenisation, 8 regional, and 242 “other” families, illustrating the richness of variation in informal Luxembourgish writing. The methodology demonstrates that distributional subword models can uncover systematic variation patterns without any supervised labeling, and that the resulting clusters are transparent, reproducible, and amenable to downstream sociolinguistic analysis.

The paper discusses several strengths: (1) elimination of costly lexicon creation; (2) simultaneous handling of form and meaning via combined cosine and n‑gram similarity; (3) provision of detailed scores and metadata (user IDs, timestamps) enabling quantitative studies of regional, temporal, or demographic trends. Limitations include reliance on empirically tuned hyper‑parameters, lack of a standardized benchmark for evaluating variant family quality, and potential exclusion of extremely rare forms due to frequency thresholds. Future work is suggested on automatic hyper‑parameter optimization, development of intrinsic evaluation metrics for variant clusters, and application of the framework to other low‑resource or highly dialectal languages.

In conclusion, the subword embedding approach presented offers a powerful, language‑agnostic tool for turning what is traditionally treated as “noise” into valuable linguistic signal, thereby advancing both computational processing and sociolinguistic understanding of variation in multilingual, low‑resource contexts.