Continuous / Dynamic Batching

1. Concept Overview & Systemic Problem

Traditional machine learning (e.g., image classification in ResNet) operates with fixed tensor sizes: you take 32 images, process them through a neural network in 10 milliseconds, and return 32 labels.

However, in the realm of generative artificial intelligence, the duration of generation is unpredictable:

• One user requests a "Yes" or "No" answer (1 token).
• Another requests to write an entire book or refactor a monolith (4000 tokens).
• With a static approach, the GPU utilizes less than 20% of its throughput, forcing clients to wait in massive queues.

Continuous / Iteration-Level Batching removes rigid boundaries between requests: inference transforms into a continuous stream of generation at the level of individual cycles.

2. Architectural Taxonomy & Mental Model

┌─────────────────────────────────────────────────────────────┐
│                 STATIC VS CONTINUOUS BATCHING               │
├─────────────────────────────────────────────────────────────┤
│ STATIC BATCHING:                                            │
│ Request 1: [Token 1] [Token 2] [END] [IDLE...] [IDLE]     │
│ Request 2: [Token 1] [Token 2] [Token 3] [Token 4] ... [1000] │
│ • GPU wastes memory and resources on empty slots            │
├─────────────────────────────────────────────────────────────┤
│ CONTINUOUS BATCHING (vLLM / Orca / TGI):                   │
│ Cycle 1: [Request A - Token 1] [Request B - Token 1] [Request C - Token 1]│
│ Cycle 2: [Request A - Token 2] [Request B - Token 2] [Request C - END]
│ Cycle 3: [Request A - Token 3] [Request B - Token 3] [Request D - START]
│ • New Request D connects instantly without idle time        │
└─────────────────────────────────────────────────────────────┘

3. Technical Pipeline & Internal Mechanics

01. Corporate Agent Hub Maintenance

A company deploys its own inference cluster with 4x H100 for 500 employees. Thanks to Continuous Batching, the server maintains a steady output of 1500 tokens/sec, servicing parallel agent requests in real-time.

02. Mass Parallel Generation of Synthetic Data

When creating a training dataset, the system sends 10,000 prompts simultaneously. Continuous Batching automatically maintains maximum GPU utilization at 99% until the queue is fully exhausted.

4. Production Engineering Scenarios

• Starvation of Long Prompt Queue: If the system prioritizes fast requests, users with large prompts (e.g., 100k context for code analysis) may wait too long in the queue. Fair-Share schedulers need to be configured.
• Preemption Imbalance: If VRAM is fully utilized and the queue requires new pages, the scheduler may need to temporarily swap out sessions of unfinished requests to system RAM.

5. Pitfalls, Common Mistakes & Security

Continuous Batching is the technological engine driving the modern artificial intelligence economy. It has significantly reduced the cost of token generation over the past years, making the launch of complex multi-agent pipelines accessible to every startup.