RAG (Retrieval-Augmented Generation)

1. Concept Overview & Systemic Problem

Attempting to "stuff" all internal company documents into the weights of a neural network through pre-training or Fine-Tuning is a fundamental engineering mistake:

1. Catastrophic Forgetting: The model may partially lose basic reasoning skills while trying to memorize specific corporate artifacts.
2. Right to be Forgotten / GDPR: If a confidential document is embedded in the model's weights, it cannot be removed without complete retraining costing hundreds of thousands of dollars.
3. Lack of Access Control (RBAC): A trained model may provide a regular employee with information about executive salaries, as data is mixed in shared weight matrices.

RAG (Retrieval-Augmented Generation) separates computational intelligence from data storage: the LLM remains an impartial "reasoning processor," while all knowledge is stored in an external database under strict engineering control.

2. Architectural Taxonomy & Mental Model

The evolution of the RAG architecture is classified into three generations:

• 1. Naive RAG: A linear pipeline: Chunking -> Embedding -> Vector DB -> Top-K Nearest Neighbors -> Prompt Context. It has a high noise level, is blind to exact matches, and suffers from hallucinations with poor text cuts.
• 2. Advanced RAG: Includes stages before and after retrieval:
  • Pre-Retrieval: query expansion, hypothetical answer generation (HyDE), and routing to different knowledge bases.
  • Retrieval: hybrid search (BM25 + HNSW vectors).
  • Post-Retrieval: filtering by metadata, context compression, and cross-encoder re-ranking.
• 3. Modular / Agentic RAG: The agent decides when it needs to search, verifies received facts (Self-RAG), adjusts the query in case of data shortages, and uses graph indexes (GraphRAG) to identify complex intersubjective relationships.

3. Technical Pipeline & Internal Mechanics

The complete production cycle of a RAG system consists of two loops:

Indexing Loop (Offline Ingestion Pipeline):

1. Parsing & Clean: Extracting text from PDF, Markdown, Notion, or SQL, cleaning it from markup.
2. Syntax-Aware Chunking: Segmenting by logical boundaries (AST / headers) with an overlap buffer.
3. Vectorization & Indexing: Generating embedding vectors and storing them in a vector DB with access metadata.

Query Loop (Online Query Pipeline):

1. Query Transformation: Converting the user question into an optimized search query.
2. Hybrid Retrieval: Simultaneously retrieving candidates through vectors and full-text index BM25.
3. Re-ranking: A cross-encoder model ranks the top-30 found chunks and retains the top-5 most relevant.
4. Grounded Generation: The LLM receives a strict prompt with inserted chunks and formulates an accurate response with mandatory source citations.

4. Production Engineering Scenarios

01. Corporate Assistant with Access Control (RBAC)

The user query is accompanied by their JWT token. The vector database filters out documents at the pre-filtering stage: WHERE team_id = 'engineering' AND access_level <= user.level. The model physically does not receive context information that the user does not have access to.

02. Technical Documentation for Engineers with Direct Links

A developer queries: "How to set up SQLite replication?" RAG returns the exact sequence of steps with tags [source: docs/replication.md#L45], allowing the engineer to click through to the source code in one action.

03. Automated Compliance Audit of Legal Contracts

The system checks a 100-page contract against a registry of corporate policies, retrieving only relevant points about force majeure and penalties.

5. Pitfalls, Common Mistakes & Security

• Knowledge Base Poisoning: An attacker uploads a file with hidden instructions (Prompt Injection) into the internal database, causing RAG to issue false instructions to all users. Verify knowledge sources and use guardrails.
• Garbage In, Garbage Out: If you upload a poorly scanned PDF with a lot of broken words into RAG, no model will be able to provide an adequate response.
• Loss of Context Between Chunks: Using overly fine-grained chunking deprives sentences of context. Always enrich chunks with parent section headers.