DeepSeek-V4.1-Flash with 763B Parameters Can Run Locally on a 12GB GPU
DeepSeek-V4.1-Flash with 763B Parameters Can Run Locally on a 12GB GPU
IST Austria researchers introduced a quantized version of DeepSeek-V4.1-Flash that can run on consumer GPUs with 12GB of VRAM.
Researchers at the Austrian institute IST Austria have introduced a quantized version of the DeepSeek-V4.1-Flash AI model with 763 billion parameters that can run locally even on a GPU with 12GB of VRAM.
The GSQ-RCO quantization method and the Strata inference engine make it possible to use the model on consumer GPUs.
According to test results:
RTX 5070 with 12GB VRAM and 64GB RAM delivers about 5 tokens per second during answer generation and up to 231 tokens per second when processing input context. RTX 3060 with 12GB VRAM and 46GB RAM reaches 8–10 tokens per second during answer generation and around 149 tokens per second when processing input context.
Running the model requires a large amount of RAM and at least 239GB of free storage, because most model weights are stored in RAM and on SSD, loading as needed.
The speed is hard to call high, and complex agentic tasks still require substantial computing resources. Still, the ability to run a 763-billion-parameter model on an ordinary computer shows significant progress in quantization and inference optimization.
Why it matters
AnalysisIt highlights progress in quantization and inference optimization, though running large AI models locally still requires substantial resources.
Discuss in community
Share your questions and insights with developers
Related in Ecosystem
Practical guides, core terminology, and agent skills
GGUF & Modern Quantization Standards
The GGUF format is a universal binary file format for storing and instantaneously loading quantized language models on CPUs and GPUs in llama.cpp, Ollama, and LM Studio.
Apple Silicon MLX Framework
A native machine learning library from Apple, designed to maximize the use of unified memory and GPU cores in M-series chips (M2/M3/M4) for running large LLMs.
How to Run Local LLMs: Setup Guide, Model Benchmarks, and Optimization
The definitive guide to running local LLMs on your hardware: hardware requirements, Ollama and Open WebUI deployment, Llama 3, Qwen 2.5, DeepSeek, and GGUF quantization.
How to Run DeepSeek-R1 Locally for Free on Mac, Windows, and Linux
Tired of the limitations of cloud services and constant payments for DeepSeek-R1 API? You are not alone. Many developers, AI users, and researchers want complete control over their data and workflows.
Kimi K3.1 Pro Successfully Generates a 3D Website with a Robotic Arm Disassembled into 2,317 Parts
Kimi K3.1 Pro handled a request to create an interactive 3D website where a robotic arm is disassembled into 2,317 individual parts.
Open-source project with 74 skills for game development
A GitHub open-source project called awesome-gamedev-agent-skills brings together 74 skills for Unity, Godot, Unreal Engine and other game engines, with automatic skill selection and support for Claude Code and Codex.