What LLMs Think When You Don't Tell Them What to Think About?

Characterizing the behavior of large language models (LLMs) across diverse settings is critical for reliable monitoring and AI safety. However, most existing analyses rely on topic- or task-specific prompts, which can substantially limit what can be observed. In this work, we study what LLMs generate from minimal, topic-neutral inputs and probe their near-unconstrained generative behavior. Despite the absence of explicit topics, model outputs cover a broad semantic space, and surprisingly, each model family exhibits strong and systematic topical preferences. GPT-OSS predominantly generates programming (27.1%) and mathematical content (24.6%), whereas Llama most frequently generates literary content (9.1%). DeepSeek often generates religious content, while Qwen frequently generates multiple-choice questions. Beyond topical preferences, we also observe differences in content specialization and depth: GPT-OSS often generates more technically advanced content (e.g., dynamic programming) compared with other models (e.g., basic Python). Furthermore, we find that the near-unconstrained generation often degenerates into repetitive phrases, revealing interesting behaviors unique to each model family. For instance, degenerate outputs from Llama include multiple URLs pointing to personal Facebook and Instagram accounts. We release the complete dataset of 256,000 samples from 16 LLMs, along with a reproducible codebase.

💡 Research Summary

The paper investigates the “top‑of‑mind” behavior of large language models (LLMs) when presented with minimal, topic‑neutral prompts that impose virtually no constraints on the generation. While most prior work relies on task‑specific or topic‑specific prompts to probe model biases and risks, this study deliberately removes such scaffolding to uncover hidden tendencies that only emerge under near‑unconstrained conditions.

Prompt Design and Generation Protocol
The authors craft 36 seed prompts covering six stylistic categories: conversational softeners, chain‑of‑thought starters, declarative statements, rhetorical questions, informative expository openings, and punctuation‑only prompts. Each prompt is short, contains no named entities or domain‑specific keywords, and is intended to be truly neutral. For each of the sixteen LLMs (four families: GPT‑OSS, DeepSeek, Llama, Qwen, spanning 3 B to 671 B parameters), the models generate 16 000 samples, yielding a total of 256 000 outputs. Generation uses temperature = 1.0 and top‑p = 0.9, and no chat system prompt or role tags are supplied, ensuring that the model’s response is driven solely by the seed prompt.

Degenerate Text Detection
A key methodological contribution is the definition and removal of “degenerate text.” A segment is labeled degenerate if (i) it contains a repeated substring of at least 10 characters, (ii) this substring repeats consecutively at least five times, and (iii) the repeated portion occupies at least 5 % of the total output length. When such a pattern is detected, the output is truncated at the first occurrence, producing a cleaned corpus for downstream analysis.

Semantic Labeling and Embedding Extraction
The cleaned texts are annotated using an LLM‑based open‑vocabulary annotator (GPT‑OSS‑120B), which assigns both a high‑level category (e.g., programming, literature) and a fine‑grained sub‑category (e.g., dynamic programming, topology). For vector representations, the authors employ Qwen3‑Embeddings‑8B, an open‑source embedding model noted for strong performance. Sensitivity analyses confirm that the main findings are robust to alternative annotators and embedding models.

Broad Semantic Coverage
Even with neutral prompts, the generated corpus spans a remarkably wide semantic space. UMAP visualizations reveal 123 well‑populated clusters, covering domains such as literature, philosophy, physics, mathematics, programming, law, finance, music, sports, cooking, agriculture, archaeology, military, and fashion. These clusters account for 98.6 % of the dataset, demonstrating that LLMs can spontaneously produce content across virtually any human knowledge area without explicit prompting.

Model‑Specific Topical Preferences
Despite the shared prompt pool, each model family exhibits distinct distributional signatures:

• GPT‑OSS – Dominates programming (27.1 %) and mathematics (24.6 %). Within programming, it favors advanced topics like dynamic programming, algorithmic design, and multi‑language code snippets. Its mathematical outputs lean toward topology, combinatorics, and higher‑level theory.
• Llama – Shows a bias toward the liberal arts: literature (9.1 %), psychology (7.6 %), philosophy (6.7 %). Its “degenerate” outputs frequently contain URLs to personal Facebook and Instagram accounts, a pattern not observed in other families.
• DeepSeek – Generates a disproportionate amount of religious content, often quoting Bible verses (e.g., Matthew, Hebrews). Sub‑category analysis shows a focus on theology and moral philosophy.
• Qwen – Produces many multiple‑choice exam‑style questions and conversational artifacts such as “let me know” or “thank you,” along with occasional Chinese language text.

Sub‑category heatmaps (Figure 3) illustrate that even within the same high‑level domain, the families specialize differently (e.g., GPT‑OSS favors Python/C++ while DeepSeek leans toward JavaScript/web development).

Degenerate Text as a Model Signature
The study uncovers systematic, family‑specific repetitive patterns that act as behavioral fingerprints:

• GPT‑OSS – Repeated code block delimiters (e.g., “\n\n’’’ \n\n’’’”).
• Qwen – Repetitive polite phrases and Chinese characters.
• Llama – Repeated personal social‑media URLs.
• DeepSeek – Repeated scriptural citations.

These patterns suggest that certain training‑data templates have been over‑learned and surface when the model is not steered by a system prompt.

Implications for Safety and Monitoring
The findings have direct relevance to AI safety and reliability. Model‑specific topical biases could lead to unintended exposure of sensitive or copyrighted material (e.g., personal URLs), while the propensity to generate advanced technical content may increase the risk of facilitating malicious code creation. The identification of degenerate signatures provides a lightweight diagnostic for continuous model monitoring: a sudden shift in repetition patterns could indicate data drift or fine‑tuning anomalies.

Contributions and Resources
The paper’s primary contributions are:

1. A novel, minimally constrained probing framework that reveals hidden generative tendencies of LLMs.
2. A large‑scale dataset of 256 k samples across sixteen state‑of‑the‑art models, publicly released on Hugging Face.
3. Comprehensive quantitative and visual analyses of topical distribution, sub‑category specialization, technical depth, and degenerate text patterns.
4. An open‑source codebase enabling reproducibility and extension to future models.

Overall, the work demonstrates that even in the absence of explicit guidance, LLMs exhibit strong, systematic preferences and idiosyncratic repetitive behaviors. These insights enrich our understanding of model internals, inform risk assessment pipelines, and open new avenues for research on prompt‑free model auditing.