Paged Attention & KV-Cache Management
A GPU memory management algorithm that segments the KV-cache of a language model into contiguous virtual pages (similar to OS kernels), eliminating fragmentation and increasing throughput by four times.
1. Concept Overview & Systemic Problem
When a server handles requests to a language model, memory is the biggest scaling bottleneck:
- The NVIDIA A100 GPU has 80 GB of VRAM. The 70B model in 4-bit quantization occupies ~38 GB.
- This leaves 42 GB for user requests.
- If memory is reserved in the traditional manner, the server can only handle 2–3 long-context requests simultaneously. Attempting to connect a fourth client results in a
CUDA Out of Memoryerror.
Paged Attention addresses this issue by transferring a 50-year-old foundational principle of operating systems — virtual paging — into GPU memory.
2. Architectural Taxonomy & Mental Model
┌─────────────────────────────────────────────────────────────┐
│ PAGED ATTENTION ARCHITECTURE │
├─────────────────────────────────────────────────────────────┤
│ 1. Logical KV Blocks (User Context Logical Blocks): │
│ [Tokens 0-15] ➔ [Tokens 16-31] ➔ [Tokens 32-47] │
├─────────────────────────────────────────────────────────────┤
│ │ │
│ ▼ Block Table (Page Table) │
│ • Logical Block 0 ➔ Physical Block 7 in VRAM │
│ • Logical Block 1 ➔ Physical Block 2 in VRAM │
│ • Logical Block 2 ➔ Physical Block 11 in VRAM │
├─────────────────────────────────────────────────────────────┤
│ 2. Physical VRAM Pages (Contiguous Free Slots): │
│ ┌─────────┬─────────┬─────────┬─────────┬─────────┐ │
│ │ Block 0 │ Block 1 │ Block 2 │ ... │ Block 11│ │
│ │ (Req B) │ (Free) │ (Req A) │ │ (Req A) │ │
│ └─────────┴─────────┴─────────┴─────────┴─────────┘ │
│ • Memory fragmentation reduced from 70% to <4% │
└─────────────────────────────────────────────────────────────┘
3. Technical Pipeline & Internal Mechanics
01. Prefix Caching / Copy-on-Write
If 50 users simultaneously query the same system prompt or PDF document, Paged Attention does not duplicate it 50 times. All requests reference the same physical KV-cache pages in VRAM. Memory is only copied when a user begins generating their own unique response (Copy-on-Write).
02. Branching Search
When an agent explores 4 options for solving a problem (Beam Search or Tree-of-Thought), the initial shared context is retained in a single instance of memory pages, saving up to 75% of VRAM during swarm computations.
4. Production Engineering Scenarios
01. Overhead of Page Tables with Short Texts
For ultra-short dialogues (1–5 tokens), the overhead of managing the block table may slightly increase dispatch time.
02. Page Cleanup on Request Cancellation
If a user disconnects, the engine must immediately return the allocated physical pages to the free block pool; otherwise, a memory leak (VRAM Memory Leak) occurs.
5. Pitfalls, Common Mistakes & Security
Paged Attention has quietly revolutionized LLM infrastructure in the same way that virtual memory transformed Unix. It has made servicing hundreds of agents on a single GPU an economically viable industry standard.
FAQ: Paged Attention & KV-Cache Management
Related terms
vLLM (High-Performance Inference Engine)
Leading open-source inference engine and LLM servicing framework that revolutionizes throughput with the PagedAttention memory virtualization algorithm and continuous batching.
Context Window
The maximum operational token capacity that a language model can simultaneously hold in the Self-Attention mechanism and KV Cache memory during a single inference request.
KV-Cache Offloading & Compression
Hardware and algorithmic methods for temporarily offloading Key-Value Cache (KV-Cache) from expensive GPU VRAM to system RAM or fast NVMe SSDs.
Continuous / Dynamic Batching
A mechanism for grouping incoming requests to neural networks at the token iteration level (Iteration-Level Scheduling), eliminating GPU idle time during parallel loads.