Prompt Caching & KV Cache Reuse

1. Concept Overview & Systemic Problem

In modern development using agents (e.g., Claude Code or Cursor), each dialogue consists of dozens of iterations. Notably, 90–95% of the information in each request is entirely static:

• System prompt with project rules (5,000 tokens).
• Schemas of 20 connected MCP tools (10,000 tokens).
• Indexed documentation and code files (40,000 tokens).

Without caching, for each developer reply, the server must reprocess all 55,000 static tokens through the transformer matrices (Prefill Phase):

1. High Financial Cost: You pay the full price for reading the same 55,000 tokens 30 times per session.
2. High Latency (High TTFT): Time to the first character of the response stretches to 5–10 seconds, killing the interactive vibe of coding.
3. GPU Resource Waste: GPU computational power is wasted on repeating deterministic calculations.

Prompt Caching eliminates this overhead: the server computes the attention state for the static prefix once, stores it in ultra-fast HBM memory, and instantly retrieves the ready state on subsequent calls.

2. Architectural Taxonomy & Mental Model

The architecture of prompt caching is built around the following conceptual layers:

• 1. Prefix Structure (Prefix Hierarchy): The prompt is designed as a nesting doll from the most static to the most dynamic: [Static System Prompt] -> [MCP Tool Definitions] -> [Cached Project Context] -> [Dynamic Turn History] -> [User Input].
• 2. Cache State Economics:
  • Cache Write: the first access or write after a prefix change. Costs 25% more than standard input, as it requires state retention in memory.
  • Cache Hit / Read: repeated access to an unchanged prefix. Provides an 80–90% discount off the base cost of input tokens.
• 3. Breakpoints: Specific markers in the request structure that inform the inference engine where the stable block that can be cached ends.
• 4. Cache Lookup Mechanisms:
  • Radix Tree / Prefix Trie: a data structure in the memory of the inference server (e.g., in vLLM / SGLang) that instantly finds the longest common chain of tokens among active sessions.

3. Technical Pipeline & Internal Mechanics

The lifecycle of a request using Prompt Caching:

1. Serialization & Hash Verification: The client sends an array of messages. The server takes the sequence of tokens from the start to the first cache_control point and computes a cryptographic hash of the prefix.
2. Trie Lookup: The GPU memory manager checks if there is a live KV cache block for this hash.
   • If found (Cache Hit) — Prefill computation is skipped, and the stored matrices $K$ and $V$ are instantly connected to the computation graph.
   • If not found (Cache Miss) — the server performs full computation and asynchronously writes the result to the buffer.
3. Dynamic Suffix Processing: The transformer computes attention exclusively for the new fragment — the last user message (e.g., 200 tokens instead of 50,000).
4. Immediate Decoding & TTL Refresh: The model begins generating a response within 200–400 milliseconds, and the cache's lifetime timer resets to a new 5-minute interval.

4. Production Engineering Scenarios

01. Multi-Step Development Sessions in Claude Code and Cursor

An agent performs a complex 15-step refactoring. Thanks to caching, the project's codebase (100,000 tokens) is cached at step 1. For steps 2–15, the user only pays for the delta of changes, saving up to 85% of the total development cost for the evening.

02. Caching Large MCP Tool Libraries

Connecting 30 complex enterprise tools requires detailed JSON schemas that take up 12,000 tokens. Caching the tool block ensures that the developer does not overpay for their description on every trivial question in the chat.

03. High-Load Corporate Assistants with Knowledge Bases

An internal support chatbot uses a single cached system document (regulations, instructions, FAQs) totaling 50,000 tokens. Thousands of employees send requests simultaneously, hitting the same cached layer on the model cluster, ensuring minimal response time.

5. Pitfalls, Common Mistakes & Security

• Dynamic Data Leakage into the Prefix (Timestamp Invalidation): The most common engineering mistake is adding a dynamic string like Current Time: 2026-09-09 12:15:30 at the beginning of the system prompt. This changes the first tokens every second and completely breaks caching for all subsequent modules. Move dynamic variables to final messages.
• Cold Start After Idle Time: If more than 5 minutes pass between developer steps, the cache evaporates. The next request is charged again at the Cache Write rate with initialization delay.
• Mixing Access Rights (Multi-Tenant Cache Bleed): On self-hosted servers (vLLM), ensure that sessions from different users do not share a prefix containing confidential personal data.