Prompt Caching Architecture & Economics

1. Concept Overview & Systemic Problem

In the era of autonomous agents, interaction with the model is multi-step:

• Step 1: Send 80k tokens of code + question.
• Step 2: Receive response + call tool + send tool result (again 80k tokens of code + new context).
• By the 10th step, the developer pays for re-reading the same 80k tokens 10 times in a row!
• Without caching, the session costs $2.50. With caching — $0.28.

Prompt Caching Economics is a new engineering discipline: designing dialogue structures so that static data is never re-computed on the GPU.

2. Architectural Taxonomy & Mental Model

┌─────────────────────────────────────────────────────────────┐
│                 PROMPT CACHING PREFIX RULES                 │
├─────────────────────────────────────────────────────────────┤
│ 1. STATIC CACHEABLE PREFIX (Must be 100% identical):       │
│    ┌─────────────────────────────────────────────────────┐  │
│    │ • Base System Prompts & Strict Agent Rules          │  │
│    │ • Project Schemas & TypeScript Definitions          │  │
│    │ • Immutable Core Documentation & Tool Specs         │  │
│    └─────────────────────────────────────────────────────┘  │
│    ➔ HIT: 90% DISCOUNT & 4x FASTER PREFILL SPEED            │
├─────────────────────────────────────────────────────────────┤
│                          │                                  │
│                          ▼ The Dynamic Append Boundary      │
├─────────────────────────────────────────────────────────────┤
│ 2. VOLATILE DYNAMIC SUFFIX (Appended only at the end):      │
│    • Current user query & conversation turns                │
│    • Dynamic tool outputs & runtime error logs              │
│    • Timestamps & ephemeral IDs                             │
└─────────────────────────────────────────────────────────────┘

3. Technical Pipeline & Internal Mechanics

01. Proper Placement of Dynamic Metadata

Anti-Pattern (Cache Death):

[System Prompt] Today: 2026-09-09 13:42:15. Your task is to code...

(The time changes every second, causing the cache to reset 100% of the time).

Correct Pattern (Eternal Cache): Time and variables are passed in the last user message:

[Static System Prompt - 40k tokens] (CACHED)
[User Message] [Timestamp: 13:42:15] Add a new button to the header.

02. Using Cache Breakpoints in Anthropic

Specify the header cache_control: {"type": "ephemeral"} on a massive block of documentation:

const messages = [
  {
    role: "system",
    content: [
      {
        type: "text",
        text: massiveCodebaseSnapshot,
        cache_control: { type: "ephemeral" }
      }
    ]
  },
  { role: "user", content: "Fix the bug in the router" }
];

4. Production Engineering Scenarios

01. Cache Management for Long Sessions

Implementing effective prompt caching can significantly reduce costs in long-running sessions, where repeated reads of static data can be avoided.

02. Dynamic Metadata Handling

Ensure that dynamic elements are appended correctly to prevent cache invalidation, maintaining efficiency and reducing unnecessary costs.

03. Cache Control Strategies

Utilize cache control headers to manage ephemeral data effectively, ensuring that only necessary data is cached while optimizing performance.

5. Pitfalls, Common Mistakes & Security

• Too Short Prefix: Most providers require a minimum text size for caching (e.g., at least 1024 tokens in Anthropic or 2048 in DeepSeek). Caching a short prompt of 100 tokens is futile.
• Cold Start Cost: The first request with a new prefix costs 25% more than a standard input token (Cache Write surcharge). Savings begin with the second and subsequent requests.