Quantization Methods for LLMs to Run a 70B Model on a Single GPU
The article discusses five quantization methods for LLMs that reduce memory requirements for large models like 70B. Each method addresses the issue of outliers at different stages.
Quantization Methods for LLMs to Run a 70B Model on a Single GPU
A 70B model in FP16 requires about 140 GB just for weights. In a 4-bit format, this volume reduces to 35 GB, which fits on a single GPU.
However, simple rounding of values does not work for large models. Approximately 0.1% of hidden dimensions contain values that can be up to 20 times larger than other values in the tensor, and they break the quantization grid for all other parameters.
Each of these five methods addresses the outlier problem at different stages:
RTN (Round-To-Nearest)
Ignores the problem.
Each weight is simply rounded to the nearest quantization level without using calibration data.
The cheapest option, but the weakest at low precision.GPTQ
Corrects the consequences of rounding.
Quantizes layer by layer, column by column, and after each step adjusts the remaining weights to compensate for the error before moving on.AWQ (Activation-aware Weight Quantization)
Protects important weights before rounding.
Identifies about 1% of weight channels that have the most significant impact and scales them to better withstand quantization.
The final model still operates entirely in standard INT4.LLM.int8()
Isolates outliers during inference.
Dimensions with outliers are executed in FP16, while the remaining 99.9% of parameters operate in INT8. The results are then combined.QAT (Quantization-Aware Training)
Addresses the problem during training.
The model is fine-tuned with quantization in mind: rounding is built into each forward pass, so the model adapts to the resulting losses before actual quantization is applied.
All five methods produce the same result — a model that operates with lower representation precision compared to the original trained version.
The difference lies only in when the outlier problem is addressed.
The visualization below summarizes these approaches well.
There is an excellent article with a detailed study of LLM quantization methods:
Read more
Why it matters
AnalysisThese quantization methods allow for reduced memory requirements for large models, making them more accessible for use on standard GPUs. Understanding these approaches is crucial for developers looking to optimize their models.
Discuss in community
Share your questions and insights with developers
Related in Ecosystem
Practical guides, core terminology, and agent skills
MIT researchers show GPT-6 Astra can improve other AI models by editing their weights
In an experiment, Astra raised Qwen3.5-0.8B's accuracy on MATH-500 from 7.9% to 48.1% by changing only 0.016% of the model's weights.
AI Radar: GPT-6.1 Terra, Gemini 4 Carbon, Mistral Large 4 and OpenAI/Anthropic pricing changes
A roundup of recent AI news: new LLM releases, OpenAI and Anthropic subscription changes, world models, video generation, and research updates.
Model Quantization
A mathematical compression technology for neural network weights and activations by transitioning from high precision (FP16/BF16) to low-bit formats (FP8, INT8, INT4, GGUF) for radical memory savings.
Part II. AI Agent Architecture: From Model to Memory and Tools
A deep engineering dive: tokenization, KV-cache, sampling determinism, the training stack, quantization, context engineering, the safety stack, production RAG, ReAct, and multi-agent swarm systems.