Beyond Chunk-Then-Embed: A Comprehensive Taxonomy and Evaluation of Document Chunking Strategies for Information Retrieval

Document chunking is a critical preprocessing step in dense retrieval systems, yet the design space of chunking strategies remains poorly understood. Recent research has proposed several concurrent approaches, including LLM-guided methods (e.g., DenseX and LumberChunker) and contextualized strategies(e.g., Late Chunking), which generate embeddings before segmentation to preserve contextual information. However, these methods emerged independently and were evaluated on benchmarks with minimal overlap, making direct comparisons difficult. This paper reproduces prior studies in document chunking and presents a systematic framework that unifies existing strategies along two key dimensions: (1) segmentation methods, including structure-based methods (fixed-size, sentence-based, and paragraph-based) as well as semantically-informed and LLM-guided methods; and (2) embedding paradigms, which determine the timing of chunking relative to embedding (pre-embedding chunking vs. contextualized chunking). Our reproduction evaluates these approaches in two distinct retrieval settings established in previous work: in-document retrieval (needle-in-a-haystack) and in-corpus retrieval (the standard information retrieval task). Our comprehensive evaluation reveals that optimal chunking strategies are task-dependent: simple structure-based methods outperform LLM-guided alternatives for in-corpus retrieval, while LumberChunker performs best for in-document retrieval. Contextualized chunking improves in-corpus effectiveness but degrades in-document retrieval. We also find that chunk size correlates moderately with in-document but weakly with in-corpus effectiveness, suggesting segmentation method differences are not purely driven by chunk size. Our code and evaluation benchmarks are publicly available at (Anonymoused).

💡 Research Summary

The paper addresses a fundamental yet under‑explored component of dense retrieval pipelines: document chunking. While recent works have introduced a variety of chunking strategies—ranging from simple structure‑based methods (fixed‑size windows, sentence‑level, paragraph‑level) to more sophisticated semantic or LLM‑guided approaches (semantic similarity splitting, proposition‑based splitting, and the LumberChunker)—they have been evaluated in isolation, on different datasets, with different embedding models, and under different evaluation metrics. This makes it impossible to draw clear conclusions about which strategies are truly superior and under what conditions.

To fill this gap, the authors propose a unified experimental framework that classifies chunking methods along two orthogonal dimensions: (1) Segmentation Method – how the document is split, and (2) Embedding‑Chunk Ordering – whether chunking occurs before embedding (pre‑embedding chunking) or after a full‑document embedding (contextualized or post‑embedding chunking). The segmentation taxonomy distinguishes between structure‑based methods (fixed‑size, sentence, paragraph) that rely on explicit textual boundaries, and semantic/LLM‑guided methods (semantic similarity, proposition, LumberChunker) that use language models to detect logical or topical shifts. The embedding ordering dimension captures the trade‑off between efficiency (pre‑embedding) and contextual completeness (post‑embedding).

The authors evaluate twelve concrete configurations (five segmentation methods × two ordering strategies, plus the LLM‑guided variants) across two retrieval tasks that represent distinct real‑world use cases:

1. In‑Document Retrieval (needle‑in‑a‑haystack) – using the GutenQA dataset (100 classic books, 3 000 QA pairs). The task requires locating a specific paragraph within a single long document. Performance is measured with Discounted Cumulative Gain at rank 10 (DCG@10).

2. In‑Corpus Retrieval – using six BEIR benchmark datasets (FiQA, ArguAna, SciDocs, TREC‑COVID, SciFact, NFCorpus). This is the classic ad‑hoc retrieval scenario where relevance is judged at the document level. The primary metric is nDCG@10.

Four embedding models are employed to test generality: Jina‑v2 (33 M parameters, efficient), Jina‑v3 (570 M, multilingual), Nomic‑v1 (137 M, strong on diverse tasks), and E5‑large (560 M, multilingual). All models support mean‑pooling, which is required for contextualized chunking. For LLM‑guided segmentation, the authors use Gemini‑2.5‑Flash (temperature 0) to ensure deterministic outputs.

Key Findings

• Task‑dependent optimality – In‑corpus retrieval consistently favors the simple structure‑based methods (fixed‑size and sentence‑based) regardless of the embedding model. The semantic and LLM‑guided methods underperform, suggesting that for large heterogeneous corpora, the overhead of sophisticated segmentation does not translate into better relevance signals. Conversely, in‑document retrieval shows a clear advantage for the LumberChunker (LLM‑guided) approach, which outperforms all others by a noticeable margin. This indicates that when the goal is to pinpoint a specific fact inside a single long text, capturing discourse‑level shifts via an LLM is beneficial.

• Effect of contextualized chunking – Applying post‑embedding (contextualized) chunking improves nDCG@10 on the BEIR datasets by roughly 3–5 % on average, confirming prior observations that preserving global context helps ad‑hoc retrieval. However, the same technique degrades DCG@10 on GutenQA by 2–4 %, likely because the pooled token representations become less discriminative for fine‑grained location tasks.

• Chunk size vs. effectiveness – Correlation analysis reveals a moderate positive relationship between average chunk length and in‑document retrieval performance (Pearson ≈ 0.45), but only a weak correlation for in‑corpus retrieval (≈ 0.12). This suggests that while larger chunks can guarantee that the target information is not split (beneficial for needle‑in‑a‑haystack), they do not substantially drive performance in standard IR where many documents compete.

• Model‑agnostic trends – The observed patterns hold across all four embedding models, indicating that the interaction between segmentation strategy and retrieval task is robust to model size, multilingual capability, and training data.

Implications for Practitioners

• For large‑scale search engines or any system that indexes millions of documents, the simplest structure‑based pre‑embedding chunking (e.g., 256‑token fixed windows or 5‑sentence groups) is both computationally cheap and empirically optimal.

• For applications that need to retrieve precise passages from a single long document (legal contracts, medical reports, long‑form articles), investing in an LLM‑guided segmentation like LumberChunker and optionally applying contextualized chunking can yield significant gains.

• When computational budget is tight, the modest improvements from contextualized chunking on BEIR may not justify the extra inference cost of large‑context models; a hybrid approach (contextualized only for high‑value queries) could be explored.

• Developers should focus less on fine‑tuning chunk size and more on aligning the segmentation philosophy with the downstream task’s granularity requirements.

Limitations & Future Work

• The study uses a single LLM (Gemini‑2.5‑Flash) for all LLM‑guided methods; performance may vary with different model families or prompt designs.

• Only text‑only documents are considered; extending the taxonomy to multimodal content (tables, figures, code) is an open challenge.

• The “post‑embedding” implementation relies on mean‑pooling; alternative pooling strategies (CLS token, attention‑weighted) could affect the trade‑off between context preservation and discriminability.

• Dynamic chunking—adjusting chunk boundaries on a per‑query basis—remains unexplored and could combine the strengths of both worlds.

In summary, the paper delivers a comprehensive, reproducible benchmark that clarifies when and why certain chunking strategies excel. It provides actionable guidance for both researchers and engineers designing dense retrieval pipelines, and it opens several promising avenues for further investigation into cost‑effective, task‑aware document segmentation.