Document Chunking Strategies

1. Concept Overview & Systemic Problem

Embedding models (e.g., text-embedding-3-large or bge-m3) have a rigidly limited input window length (from 512 to 8192 tokens). Even if a model can technically accept a large PDF document in its entirety, it compresses all information into a single fixed-dimension vector (e.g., 1536 numbers).

Without granular chunking:

1. Semantic Dilution (Vector Dilution): A vector from a 50-page document becomes so general that it loses details of specific functions or configurations.
2. Inability to Retrieve Accurately in RAG: The search system pulls the entire massive document into the context of the generative LLM, instantly exhausting the token budget.
3. "Killing" Structure with Naive Cutting: Fixed character-based splitting breaks words in half, separates function headers from their implementations, and destroys tables.

Chunking is a cornerstone of RAG engineering: it transforms monolithic raw text into a collection of information-rich, interrelated semantic blocks.

2. Architectural Taxonomy & Mental Model

Depending on the data structure, four architectural levels of chunking are identified:

• 1. Fixed-Size Sliding Window Chunking: The simplest approach: text is sliced into blocks of $N$ tokens with an overlap of $M$ tokens. Fast, but often disrupts logical paragraph boundaries.
• 2. Recursive Character Splitting: A hierarchical approach (LangChain/LlamaIndex standard). The algorithm attempts to cut the text first by double line breaks (\n\n), if the piece is too large — by single line breaks (\n), then by periods (. ), and only as a last resort by spaces.
• 3. Semantic / Embedding-Distance Chunking: The text is broken into individual sentences. A vector is generated for each, and cosine similarity is computed between adjacent sentences. The chunk boundary is established at points of sharp semantic change (topic shifts).
• 4. Structural AST/Markdown Chunking: Considers the document's syntax tree: it breaks Markdown strictly by header levels (#, ##, ###), preserving breadcrumbs, while program code is chunked by class and function nodes using Tree-sitter.

3. Technical Pipeline & Internal Mechanics

The professional chunking pipeline consists of four stages:

1. Document Ingestion & Pre-cleaning: Removal of invisible special characters, unification of line breaks (\r\n -> \n), and normalization of Unicode encoding (NFC).
2. Boundary Detection: The parser builds a syntax tree (AST) or identifies safe cut points (Markdown sections, empty lines between functions).
3. Window Formatting & Overlap Calculation: Text is grouped into blocks of 500–700 tokens. A buffer overlap (10–15%) from the previous chunk is added to the start of each subsequent chunk.
4. Metadata Enrichment & Context Injection: Each chunk is prefixed with metadata: [File: api/auth.ts | Class: AuthService | Method: validateToken] This ensures that the fragment's vector will be found even for queries where the method name is not mentioned in its body.

4. Production Engineering Scenarios

01. Indexing Codebase Repository via Tree-sitter

Each TypeScript file is parsed into a syntax tree. A separate chunk is formed for each exported interface and each function. If a function spans 20 lines, it becomes a standalone chunk along with its JSDoc comment and a list of imported types.

02. Processing Complex Tables and Financial Reports

Standard slicing breaks table rows, stripping numbers of meaning. Professional chunking converts each table row into a self-contained text line: "In 2024, the Cloud division's revenue was $35B, a 15% increase from 2023," forming vectors with high search accuracy.

03. Parsing API Documentation (OpenAPI / Swagger)

Each route (e.g., POST /v1/payments) is highlighted as a separate chunk containing the path, HTTP method, parameter descriptions, sample responses, and error codes, allowing the agent to generate client code flawlessly.

5. Pitfalls, Common Mistakes & Security

• Dangling Pronouns Issue: If the sentence "This function requires admin rights" ends up in chunk 2, while the function name remains in chunk 1, the model will be unable to respond to the query. Always use metadata with parent context enrichment (Parent Document Retrieval).
• Vector Database Bloat from Excessive Overlap: Overlap exceeding 25% leads to information duplication in the database, increasing indexing costs and overloading identical fragments in top results.
• Loss of Lineage Context: A chunk without file name and line number turns into nameless noise, complicating the generation of accurate references in agent responses.