TTFT vs TPS (Inference Latency Metrics)
Two key engineering metrics for inference performance: Time To First Token (latency to response initiation) and Tokens Per Second (throughput of code generation).
1. Concept Overview & Systemic Problem
When evaluating the performance of an AI model, novice developers often look at a single overall speed metric: "It generates quickly." However, in real production, latency is divided into two distinct hardware phases:
- Prefill Phase: The model must "read" an input prompt of 50,000 tokens of code. This is a compute-intensive operation that determines TTFT (Time To First Token).
- Decode Phase: The model generates new code sequentially, token by token. This is a memory-bound operation that determines TPS (Tokens Per Second).
2. Architectural Taxonomy & Mental Model
[ USER SENDS REQUEST ]
│
▼
┌───────────────────────────────────────────────────────────────────────┐
│ 1. PREFILL PHASE (Parallel processing of the entire input prompt) │
│ • GPU FLOPs utilized at 100% │
│ • Time to first character: ➔ [ TTFT = 420 ms ] │
└───────────────────────────────────┬───────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────────┐
│ 2. DECODE PHASE (Sequential generation of tokens) │
│ • Memory Bandwidth utilized at 100% │
│ • Streaming rate: ➔ [ TPS = 85 tokens/sec ] │
└───────────────────────────────────┬───────────────────────────────────┘
│
▼
[ FULL RESPONSE COMPLETED ]
3. Technical Pipeline & Internal Mechanics
01. Optimizing Interactive Assistant in IDE
Developers expect real-time code suggestions (Inline Completion). If TTFT exceeds 300 ms, developers continue typing themselves, rendering the suggestion useless. For this role, models with minimal TTFT (Gemini Flash or local 3B models) are selected.
02. Choosing a Provider for Codebase Refactoring
For the task of writing 500 lines of code in the background, a model with TTFT = 5 seconds and TPS = 120 tokens/sec will complete in 9 seconds, while a model with instant start (TTFT = 0.2s) but slow TPS = 25 tokens/sec will require 20 seconds.
4. Production Engineering Scenarios
01. Misleading Average TPS
Providers often advertise peak TPS for short responses. As the output text lengthens (KV cache grows), TPS can drop by 30–40%.
02. Impact of Queue Latency
If the server is overloaded with requests, high TTFT may be caused not by model slowness but by a request waiting 2 seconds in the scheduler's queue. Always separate Queue Time from pure GPU Prefill Time.
5. Pitfalls, Common Mistakes & Security
TTFT and TPS are the engineering coordinates for model selection architecture. Understanding the difference between Prefill and Decode phases allows for building systems that feel instantaneous to users in the interface while maximizing generation speed for autonomous agents.
FAQ: TTFT vs TPS (Inference Latency Metrics)
Related terms
Generation Speed (TPS / TTFT / Latency)
Key engineering performance metrics for language models: Time to First Token (response time to input context) and Tokens Per Second (streaming output text generation speed).
vLLM (High-Performance Inference Engine)
Leading open-source inference engine and LLM servicing framework that revolutionizes throughput with the PagedAttention memory virtualization algorithm and continuous batching.
Gemini Flash & Pro (Google Gemini)
A family of multimodal models from Google DeepMind that combines a record context window (up to 2 million tokens), extreme generation speed (over 150 tokens/sec), and native perception of video and audio.
Sampling Parameters (Temperature, Top-p, Min-p)
Mathematical hyperparameters of stochastic decoding (Temperature, Top-P, Min-P, Penalties) that govern the probability distribution for selecting the next token, defining the model's level of determinism, accuracy, and creativity.
Related Ecosystem Resources
Explore practical workflows, core terminology, and engineering tools
Generation Speed (TPS / TTFT / Latency)
Key engineering performance metrics for language models: Time to First Token (response time to input context) and Tokens Per Second (streaming output text generation speed).
Prompt Caching & KV Cache Reuse
A technology utilized by modern inference engines and cloud APIs (Anthropic, OpenAI, DeepSeek, vLLM) that stores precomputed attention matrices (KV Cache) of static prefixes, reducing processing costs by 80–90% and decreasing time to first token (TTFT) by 4–8 times.
Monitoring AI Agents: Metrics, Tracing, and Root Causes of Failures
Standard server and application APM fails for agents: an agent can return HTTP 200, burn through tokens, and still give the user an incorrect answer or get stuck in a loop.
Anthropic releases FlashProve in open source, a tool for provable inference
Anthropic published a prototype of an open cryptographic protocol called FlashProve that lets anyone verify a model response without access to model weights.
Anthropic introduces FlashProve prototype for provable AI inference
Anthropic has presented FlashProve, a prototype system that lets developers mathematically verify that an answer came from a specific AI model without exposing the model weights.
How to Build a Custom Design System in Claude Design: From Moodboard to Animated Mascot
Complete practical guide to building a brand design system in Claude Design: visual references, 4 diverging concepts, tokenization, stress-testing, and animated SVG mascot design.