DeepSeek has open-sourced DeepInfer, the high-performance inference framework that powers its own large-scale production services, publishing the code on GitHub on October 20 under the Apache 2.0 licence. The release is the first time the Chinese lab has shared the whole engine behind its low-cost API, rather than individual libraries, and it arrives three days after the lab shipped its V4.1-Flash-1017 model update — the one model DeepInfer currently supports.
DeepSeek announced the release on its official X account, describing DeepInfer as its in-house inference framework, now fully open source, and saying it powers DeepSeek-V4.1-Flash at scale with high throughput and low latency. The GitHub repository itself describes DeepInfer in the same terms and had drawn thousands of stars within hours of the announcement.
DeepInfer: What The Release Contains
The framework is built for offline inference with the core features DeepSeek uses for serving at scale. Its supported features include optimised kernels and a model and sampler runtime, attention data parallelism and balanced context parallelism, mixture-of-experts expert parallelism with load balancing, asynchronous scheduling through a zero-overhead batch scheduler, batch-invariant inference, deterministic sampling and structured outputs constrained to a regex, a JSON schema or a grammar.
The kernel layer is DeepSeek’s own stack: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang kernels. A system called DSpark accelerates decoding by drafting tokens ahead and using confidence estimates to schedule when those drafts are verified, while a bounded replay mechanism for the decoder reduces prefill work by replaying only a limited suffix of earlier context.
DeepInfer: Hardware, Requirements And Performance
DeepInfer supports exactly two hardware families: NVIDIA SM100 GPUs, the Blackwell data-centre generation, running CUDA 13, and Huawei Ascend 950 NPUs, running CANN 9.2. It requires Linux, GCC 13 or later and Python 3.12. On first run it downloads the DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face, the model the lab released on October 17 alongside its new search API.
DeepSeek’s published benchmarks were measured on four SM100 GPUs for prefill and 64 for decode, each within one scale-up domain. Reporting on the release puts the framework’s production figures at 155 trillion tokens per day per cluster, an input cache hit rate of 96 percent, average time to first token of around 400 milliseconds, and generation at 230 tokens per second, reaching up to 300 tokens per second for coding tasks. Batch-invariant inference and deterministic sampling, the features that keep outputs identical regardless of batch composition, are available on NVIDIA hardware only.
DeepInfer: What Is Deliberately Missing
DeepSeek is explicit that this is not a complete inference service. The README lists three pieces a full deployment still needs that the release does not include: a KV cache system, prefill/decode disaggregation — where it notes most of the required engine functionality is already implemented — and load balancing across data-parallel replicas for both prefill and decode. Teams hoping to reproduce DeepSeek’s pricing will still have to supply those layers themselves or borrow them from established serving projects.
The codebase’s lineage is acknowledged in the repository: DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and NVIDIA’s TensorRT-LLM. The release also reverses a position the lab set out in April 2025, when it said it had considered open-sourcing its full inference engine and decided against it because the code was a heavily customised early fork, tightly coupled to its internal cluster tools. A companion tokenizer tool, deepseek-recipe, was reported alongside the release to keep self-hosted token encoding aligned with the official API.
Why It Matters
DeepInfer is the central entity here, and its attributes explain DeepSeek’s pricing better than any model launch could: the lab’s cheap API has always rested on inference efficiency, and it has now published the production stack behind that efficiency for anyone to run, modify and sell. Its neighbours feel the effect immediately. Self-hosters and enterprise teams gain a reference engine tuned for DeepSeek models; NVIDIA keeps a first-class place in a Chinese lab’s flagship release, while Huawei’s Ascend 950 is treated as an equal target — notable software support for a chip ecosystem still building its CUDA alternative. The timing also sharpens the policy contrast: the release landed a day after the US order requiring civilian agencies to remove DeepSeek, which covers third-party products built on DeepSeek implementations, leaving US vendors who build on the open code with a compliance question the order does not answer. For readers new to the lab, our explainer on what DeepSeek is sets out how it reached this position.
Background
DeepSeek, the Hangzhou lab backed by the High-Flyer hedge fund, upended AI pricing in January 2025 when its R1 reasoning model matched frontier rivals at a fraction of their cost, a feat widely credited to inference optimisation as much as to model design. Since then it has open-sourced a steady stream of the supporting machinery — attention and matrix libraries, parallelism tools and now the engine itself — while its models face widening restrictions in Western government systems.
Source: DeepSeek, DeepInfer on GitHub; MadRobot; Crypto Briefing
0 Comments