DeepSeek has open-sourced DeepInfer, the inference framework that runs its production services. The DeepSeek DeepInfer inference framework is the engine behind DeepSeek-V4.1-Flash. It ships under the Apache 2.0 license. DeepSeek also released deepseek-recipe, its MIT-licensed token encoding and decoding layer. For developers, this is a rare look at a frontier lab’s real serving stack.
TL;DR
- Size / scope: 1 supported model (DeepSeek-V4.1-Flash), 1M-token context window per the model card.
- Runs on: NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs. Benchmarks used 4 GPUs for prefill and 64 for decode.
- Performance: Production-grade throughput; DeepSeek says 1 cluster serves 155 trillion tokens per day.
- Best: Above 25,000 output tokens/s/GPU at 256 requests per rank (EP64, 131,072 context).
- Worst: Offline inference only. KV cache system, P/D disaggregation and load balancing are not included.
What is DeepInfer?
DeepInfer is a high-performance inference engine from DeepSeek for its own models. According to it’s GitHub README, it powers DeepSeek’s large-scale production services. This release covers offline inference with the core features needed at scale. It was originally built primarily on vLLM, with ideas from SGLang and TensorRT-LLM.
The deployment Q&A gives production context. A single cluster serves 155 trillion tokens per day. DeepSeek reports overall revenue of US$38 per GPU-hour at valley pricing.
How does DeepInfer work?
The engine stacks DeepSeek’s kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. On top sit 5 system-level ideas.
Parallelism: Attention DP plus Wide-EP
Attention uses data parallelism. The MoE layers use expert parallelism with an expert-parallel load balancer (EPLB). Decode runs EP64 to EP128 inside a supernode. Prefill uses the smallest EP that memory allows, typically EP2 to EP8.
DSpark: confidence-scheduled speculative decoding
A draft model proposes 5 tokens per step. A Confidence Head predicts each draft token’s acceptance rate. The scheduler then verifies only the most valuable prefix. Per the Q&A, this cuts verified draft tokens by about 30% (5.00 to 3.2 to 4.4). Accepted length barely moves (3.29 vs 3.20).
SWA bounded replay
On a cache hit, DeepInfer recomputes only the last 128 tokens of sliding-window attention state. That makes recomputation cost constant, regardless of context length. In the decoder layers, DeepSeek says this nearly halves total prefill compute.
Asynchronous scheduling, determinism and structured output
CPU batch scheduling overlaps with GPU execution, so the GPU never idles between steps. Batch-invariant inference and deterministic sampling are available on NVIDIA only. Structured outputs support regex, JSON schema and EBNF grammars.
‘});tw.innerHTML=h;var f=tw.querySelectorAll(‘.fill’);setTimeout(function(){f.forEach(function(e,k){e.style.height=(conf[k]*44)+’px’})},40);
R.querySelector(‘#mtpdi-v’).textContent=mode==’fixed’?’5.00′:’3.2 to 4.4′;R.querySelector(‘#mtpdi-a’).textContent=mode==’fixed’?’3.29′:’3.20′;R.querySelector(‘#mtpdi-g’).textContent=mode==’fixed’?’baseline’:’+4% to 7%’;}
R.querySelectorAll(‘.seg button’).forEach(function(b){b.addEventListener(‘click’,function(){mode=b.dataset.m;R.querySelectorAll(‘.seg button’).forEach(function(x){x.classList.toggle(‘on’,x==b)});renderToks()})});
function bars(){var b=R.querySelectorAll(‘.bf’);b.forEach(function(e){e.style.width=”0″});setTimeout(function(){b.forEach(function(e){e.style.width=e.dataset.w+’%’})},60)}
layersIn();renderToks();
window.addEventListener(‘load’,resize);window.addEventListener(‘resize’,resize);setTimeout(resize,300);
})();
