DeepSeek has open-sourced DeepInfer, a high-performance inference framework specifically engineered to support large-scale production services for DeepSeek models. While the current release focuses on offline inference, it includes the core features necessary for scaling. The framework is designed to work with specific hardware, including NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It was developed with inspiration from existing frameworks like vLLM, SGLang, and TensorRT-LLM.

DeepInfer project logo and title on a GitHub repository page
The DeepInfer repository provides the open-source framework for high-performance inference.github.com

This image represents the official DeepInfer repository where the framework is hosted.

The framework introduces several specialized optimizations to improve throughput and latency. These include optimized kernels such as FlashMLA, DeepGEMM, and DeepEP, alongside advanced parallelism techniques like Attention DP (Data Parallelism), balanced CP (Context Parallelism), and MoE (Mixture of Experts) EP (Expert Parallelism) with EPLB. It also features DSpark and confidence-scheduled verification, which use draft tokens to accelerate decoding. Other technical features include asynchronous scheduling to overlap CPU batching with device execution, batch-invariant inference (for NVIDIA hardware), and deterministic sampling to ensure consistent outputs for identical input parameters.

A Pareto frontier chart showing prefill input throughput versus latency
A Pareto chart illustrating the trade-offs between input throughput and latency during the prefill stage.github.com

This figure visualizes the performance characteristics of the prefill process mentioned in the technical optimizations.

A Pareto frontier chart showing decode output throughput versus per-user speed
A Pareto chart comparing output throughput and per-user speed during the decoding phase.github.com

This chart demonstrates the decoding performance metrics supported by the framework's optimizations.

Despite its high-performance capabilities, the current open-source release has some limitations regarding a complete production inference service. It does not yet include a KV (Key-Value) cache system or VE/Prefill/Decode disaggregation. Additionally, while it provides the core engine functionality, it lacks built-in load balancing across data-parallel replicas for both prefill and decode stages. Users looking to deploy the framework must account for these missing components when building a full-scale service.