Skip to main content

RAG (Retrieval-Augmented Generation)

An architectural pattern for corporate AI that dynamically enriches the model's context window with relevant verified knowledge from external repositories (vector databases, graphs, full-text indexes) before generating the final response.

1. Concept Overview & Systemic Problem

Attempting to "stuff" all internal company documents into the weights of a neural network through pre-training or Fine-Tuning is a fundamental engineering mistake:

  1. Catastrophic Forgetting: The model may partially lose basic reasoning skills while trying to memorize specific corporate artifacts.
  2. Right to be Forgotten / GDPR: If a confidential document is embedded in the model's weights, it cannot be removed without complete retraining costing hundreds of thousands of dollars.
  3. Lack of Access Control (RBAC): A trained model may provide a regular employee with information about executive salaries, as data is mixed in shared weight matrices.

RAG (Retrieval-Augmented Generation) separates computational intelligence from data storage: the LLM remains an impartial "reasoning processor," while all knowledge is stored in an external database under strict engineering control.

2. Architectural Taxonomy & Mental Model

The evolution of the RAG architecture is classified into three generations:

  • 1. Naive RAG: A linear pipeline: Chunking -> Embedding -> Vector DB -> Top-K Nearest Neighbors -> Prompt Context. It has a high noise level, is blind to exact matches, and suffers from hallucinations with poor text cuts.
  • 2. Advanced RAG: Includes stages before and after retrieval:
    • Pre-Retrieval: query expansion, hypothetical answer generation (HyDE), and routing to different knowledge bases.
    • Retrieval: hybrid search (BM25 + HNSW vectors).
    • Post-Retrieval: filtering by metadata, context compression, and cross-encoder re-ranking.
  • 3. Modular / Agentic RAG: The agent decides when it needs to search, verifies received facts (Self-RAG), adjusts the query in case of data shortages, and uses graph indexes (GraphRAG) to identify complex intersubjective relationships.

3. Technical Pipeline & Internal Mechanics

The complete production cycle of a RAG system consists of two loops:

Indexing Loop (Offline Ingestion Pipeline):

  1. Parsing & Clean: Extracting text from PDF, Markdown, Notion, or SQL, cleaning it from markup.
  2. Syntax-Aware Chunking: Segmenting by logical boundaries (AST / headers) with an overlap buffer.
  3. Vectorization & Indexing: Generating embedding vectors and storing them in a vector DB with access metadata.

Query Loop (Online Query Pipeline):

  1. Query Transformation: Converting the user question into an optimized search query.
  2. Hybrid Retrieval: Simultaneously retrieving candidates through vectors and full-text index BM25.
  3. Re-ranking: A cross-encoder model ranks the top-30 found chunks and retains the top-5 most relevant.
  4. Grounded Generation: The LLM receives a strict prompt with inserted chunks and formulates an accurate response with mandatory source citations.

4. Production Engineering Scenarios

01. Corporate Assistant with Access Control (RBAC)

The user query is accompanied by their JWT token. The vector database filters out documents at the pre-filtering stage: WHERE team_id = 'engineering' AND access_level <= user.level. The model physically does not receive context information that the user does not have access to.

02. Technical Documentation for Engineers with Direct Links

A developer queries: "How to set up SQLite replication?" RAG returns the exact sequence of steps with tags [source: docs/replication.md#L45], allowing the engineer to click through to the source code in one action.

03. Automated Compliance Audit of Legal Contracts

The system checks a 100-page contract against a registry of corporate policies, retrieving only relevant points about force majeure and penalties.

5. Pitfalls, Common Mistakes & Security

  • Knowledge Base Poisoning: An attacker uploads a file with hidden instructions (Prompt Injection) into the internal database, causing RAG to issue false instructions to all users. Verify knowledge sources and use guardrails.
  • Garbage In, Garbage Out: If you upload a poorly scanned PDF with a lot of broken words into RAG, no model will be able to provide an adequate response.
  • Loss of Context Between Chunks: Using overly fine-grained chunking deprives sentences of context. Always enrich chunks with parent section headers.
/ Frequently Asked QuestionsSchema.org FAQPage

FAQ: RAG (Retrieval-Augmented Generation)

Fine-Tuning trains the model on a new style or syntax but is an unreliable method for memorizing exact facts (high risk of hallucinations) and requires costly retraining with every file change. RAG updates instantly with a simple database row update, allows for user access control (RBAC), and provides 100% clickable links to original sources.
/ Internal links

Related terms

All terms
Prompts & RAG

Vector Databases (Vector DBs & ANN Search)

Specialized DBMS and extensions (Qdrant, pgvector, Milvus, Chroma, Turso) optimized for storing millions of high-dimensional vectors and ultra-fast Approximate Nearest Neighbors (ANN) search.

Read term
Prompts & RAG

Vector Embeddings (Dense Embeddings)

A mathematical projection of text, code, or multimodal data into a dense, multidimensional numerical vector, where the angle and geometry between coordinates reflect their semantic affinity.

Read term
Prompts & RAG

Document Chunking Strategies

A methodology for decomposing massive documents and codebases into information-rich, self-contained fragments (chunks) for generating vector embeddings and precise retrieval in RAG systems.

Read term
Prompts & RAG

Hybrid Search (Dense + Sparse Search)

The retrieval architecture in modern RAG systems combines semantic vector search (Dense Embeddings) with classical keyword-based full-text indexing (Sparse/BM25) through rank fusion algorithms (RRF).

Read term