DeepSeek releases DeepInfer, an open-source inference framework
DeepSeek has published DeepInfer, an Apache-2.0 inference framework used in its production services, with optimized kernels, MoE parallelism, speculative decoding, deterministic mode, and structured outputs.
DeepSeek releases DeepInfer, an open-source inference framework
DeepSeek has published DeepInfer, the inference framework behind its production services. It is released under the Apache-2.0 license.
The core of the project includes DeepSeek's own optimized kernels (FlashMLA, DeepGEMM, DeepEP, and others), parallelism designed for MoE architectures, and an asynchronous scheduler that prepares the next batch while the device is processing the current one.
Key features include:
- Speculative decoding. The model generates several draft tokens and then validates them together, deciding which tokens can be included in the final answer and which need to be regenerated. This makes inference faster and cheaper.
- Deterministic debugging mode. Unlike normal model operation, the same inputs and parameters always produce the same result regardless of load or batch composition. This is useful for debugging pipelines and evaluations built on top of LLMs.
- Structured outputs. Generation can be constrained to JSON, regular expressions, or EBNF.
Separately, DeepSeek released deepseek-recipe under the MIT license. It describes how a client request is converted into a format understood by the model and how the model's response is converted back into what the client expects. This conversion layer often breaks self-hosted open-weight deployments. A Python binding is also available.
Why it matters
AnalysisThe release gives developers access to production-grade inference optimizations used by DeepSeek, potentially improving performance, reliability, and cost efficiency for open-weight model hosting.
Discuss in community
Share your questions and insights with developers
Related in Ecosystem
Practical guides, core terminology, and agent skills
DeepSeek opens source code for DeepInfer inference engine
DeepInfer is an open-source inference engine for large-scale V4.1-Flash serving that speeds up request handling and lowers costs.
DeepSeek 4.1-1017 nearly matches Opus 5
DeepSeek released 4.1-1017, an upgraded model for coding, agents, 3D and vision that nearly matches Opus 5 while remaining cheap and open-weight.
DeepSeek (Chinese Breakthrough in Open AI)
A series of groundbreaking open models from the Chinese lab DeepSeek (V3, R1). Demonstrates top-tier coding and reasoning capabilities at a cost 10–20 times lower than Western commercial counterparts.
Open Weights vs Truly Open Source AI
A legal and engineering analysis of the fundamental difference between accessible numerical weights of models (Llama, DeepSeek) and fully open projects with source data, code, and architecture (OSI Standard).
Jev and System One AI: A Complete Practical Guide to the Decision Model from TypeSafe
A deep engineering breakdown of the flagship Jev model: System One architecture versus LLMs, Choice/Score/Noul primitives, request parallelism, confidence mathematics, video use cases, code, and an overview of local open-weight alternatives (Laya, GLiNER2.5, CLM-8B).
Part I. AI Agent Architecture: The Foundations
The eleven foundational building blocks of modern autonomous agents: model taxonomy, the agentic runtime loop, advanced prompting, distillation, skills architecture, memory systems, function calling, and FinOps.