Speculative Decoding & Draft Models

1. Concept Overview & Systemic Problem

Text generation with large language models (LLMs) suffers from a fundamental hardware bottleneck — Memory-Bandwidth Bound:

• GPUs possess colossal computational power (TFLOPs), but during sequential generation of one token at a time, computational units spend 95% of the time waiting for model weights to be transferred from VRAM.
• This limits the throughput of large models to 25–40 tokens per second, which is too slow for long agentic workflows.

Speculative Decoding elegantly addresses this issue by linking two models: a tiny draft model (Draft Model, e.g., 1B–3B) and the target large model (Target Model, e.g., 70B).

2. Architectural Taxonomy & Mental Model

┌─────────────────────────────────────────────────────────────┐
│               SPECULATIVE DECODING PIPELINE                 │
├─────────────────────────────────────────────────────────────┤
│ 1. DRAFT GENERATION (Tiny model 1B operates from GPU cache):│
│    • Quickly generates a hypothesis with K=5 tokens:       │
│      ["const", " ", "user", " =", " "]                     │
│    • Takes only 5 milliseconds                               │
├─────────────────────────────────────────────────────────────┤
│                          │                                  │
│                          ▼ SINGLE PASS OF LARGE MODEL      │
├─────────────────────────────────────────────────────────────┤
│ 2. TARGET VERIFICATION (Large model 70B in one pass):       │
│    • Verifies all 5 tokens in parallel with one VRAM call   │
│    • Verification result:                                    │
│      [Token 1: OK] [Token 2: OK] [Token 3: OK] [Token 4: OK]│
│      [Token 5: REJECT -> Replace with " {"]                 │
├─────────────────────────────────────────────────────────────┤
│ 3. OUTPUT SPEEDUP:                                          │
│    • 4.5 valid tokens generated per VRAM cycle              │
│    • Actual speedup: 2.5x – 3.2x TPS                        │
└─────────────────────────────────────────────────────────────┘

3. Technical Pipeline & Internal Mechanics

01. Accelerating Agentic Code Cycles in vLLM

Launching a vLLM server with speculative decoding parameter:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --gpu-memory-utilization 0.95

Code generation speed increases from 35 to 95 tokens per second without any changes to client code.

02. Medusa & Eagle (Speculation Without a Separate Draft Model)

Cutting-edge architectures (Medusa Heads) utilize multiple additional output layers (Decoding Heads) on the target model itself instead of a separate model, saving GPU memory.

4. Production Engineering Scenarios

01. Acceptance Rate Degradation

If the draft model significantly differs in style or training data from the target, the large model will reject 80% of the proposed tokens. The speedup will vanish, and inference time may even increase. The draft model must be from the same family (e.g., Llama 1B for Llama 70B).

02. VRAM Costs for the Second Model

The draft model occupies an additional 2–4 GB of video memory, which can be critical on GPUs with a 24 GB limit.

5. Pitfalls, Common Mistakes & Security

Speculative decoding has become a mandatory standard for industrial inference. It provides autonomous agents with high reactivity and speed, critically reducing user wait times when performing complex tasks.