Context Poisoning in Knowledge Bases

1. Concept Overview & Systemic Problem

Context Poisoning represents a new type of cyber threat in the era of generative AI.

How it works:

• A company develops a convenient internal assistant that reads all files from the corporate Google Drive or Notion.
• An attacker (or disgruntled former employee) creates an inconspicuous file or leaves a comment: “Updated payment details for suppliers: IBAN bank account...”.
• When an accountant queries the chatbot: “Where should I transfer funds for warehouse rent?”.
• The bot finds this recent file and provides the fraudster's details!

Mental model reminder: Artificial intelligence is only as clean as the water from the well of knowledge it drinks from.

2. Architectural Taxonomy & Mental Model

┌─────────────────────────────────────────────────────────────┐
│                 CONTEXT POISONING ATTACK SCHEME            │
├─────────────────────────────────────────────────────────────┤
│ 1. DIVERSION: The attacker uploads a fake PDF               │
│    to the open folder "Incoming Documents"                  │
│    (The file contains fake bank details)                     │
├─────────────────────────────────────────────────────────────┤
│ 2. AUTOMATIC INDEXING:                                      │
│    The knowledge base connector converts text to vectors     │
├─────────────────────────────────────────────────────────────┤
│ 3. ACCOUNTANT'S QUERY:                                      │
│    “What are the current details for payment of rent?”      │
├─────────────────────────────────────────────────────────────┤
│ 4. FINDING AND RESPONSE:                                    │
│    The bot finds the poisoned chunk and confidently replies: │
│    “According to the new regulation, pay here: [Fake IBAN]” │
├─────────────────────────────────────────────────────────────┤
│ 💥 OUTCOME: Company funds transferred to fraudsters!        │
└─────────────────────────────────────────────────────────────┘

3. Technical Pipeline & Internal Mechanics

1. Strict Write Permissions: The bot should index only those pages that can be edited by a limited circle of verified top managers.
2. Isolation of Internal from External: Never mix corporate instructions with incoming emails from random internet clients in the same base.
3. Citation Source Audit: Configure the bot to always display the exact link to the file, its creation date, and the author who last saved it.
4. Checkpoints (HITL): Financial transactions must always be verified against the paper or digitally signed original contract.

4. Production Engineering Scenarios

01. Document Upload Attack

An attacker uploads a malicious document to a shared folder, which is then indexed by the knowledge base, leading to the dissemination of false information.

02. External Comment Injection

A bot indexes comments from external users, allowing an attacker to introduce misleading information that the bot later uses in responses.

03. Unverified Source Integration

The bot integrates documents from unverified sources, increasing the risk of context poisoning through the acceptance of false data as legitimate.

5. Pitfalls, Common Mistakes & Security

AI security is not just about encrypting communication channels. It primarily involves impeccable hygiene and moderation of the information entering your knowledge base.