"Documentation Index\n\nFetch the complete documentation index at: /llms.txt\n\nUse this file to discover all available pages before exploring further.\n\nSkip to main content\nSGLang Documentation home page\nSearch...\n⌘K\nAsk Assistant\nGet Started\nUser Guide\nHardware\nCookbook\nSGLang Diffusion\nBasic Usage\nBasic Usage\nOpenAI-Compatible APIs\nAnthropic-Compatible API\nOllama-Compatible API\nOffline Engine API\nSGLang Native APIs\nSampling Parameters\nAdvanced Features\nAdvanced Features\nServer Arguments\nHyperparameter Tuning\nAttention Backend\nHiSparse: Hierarchical Sparse Attention\nSpeculative Decoding\nAdaptive Speculative Decoding\nStructured Outputs\nStructured Outputs For Reasoning Models\nTool Parser\nReasoning Parser\nQuantization\nQuantized KV Cache\nDP, DPA and SGLang DP Router\nExpert Parallelism\nLoRA Serving\nPD Disaggregation\nEPD Disaggregation\nPipeline Parallelism for Long Context\nHierarchical KV Caching (HiCache)\nQuery VLM with Offline Engine\nDP for Multi-Modal Encoder in SGLang\nCuda Graph for Multi-Modal Encoder in SGLang\nBreakable CUDA Graph\nPiecewise CUDA Graph\nSGLang Model Gateway\nllm-d\nDeterministic Inference\nObservability\nModel Loading\nLoading Models from Object Storage\nCheckpoint Engine Integration\nSGLang for RL Systems\nSupported Models\nSupported models\nText Generation\nRetrieval and Ranking\nSpecialized Models\nExtending SGLang\nDeveloper Guide\nDeveloper Guide\nContribution Guide\nDevelopment\nBenchmarking\nEvaluating New Models with SGLang\nMSProbe Debugging Guide\nReferences\nReferences\nTroubleshooting and Frequently Asked Questions\nEnvironment Variables\nProduction Metrics\nProduction Request Tracing\nDeployment\nCustom Chat Template\nFrontend Language\nCookbook\nPost-Training Integration\nNightly precision regression\nAdvanced Features\nPD Disaggregation\nCopy page\n\nWhy and What is PD Disaggregation?\nLarge Language Model (LLM) inference comprises two distinct phases: Prefill and Decode. The Prefill phase is computation-intensive, processing the entire input sequence, while the Decode phase is memory-intensive, managing the Key-Value (KV) cache for token generation. Traditionally, these phases are handled within a unified engine, where combined scheduling of prefill and decode batches introduces inefficiencies. To address these challenges, we introduce Prefill and Decoding (PD) Disaggregation in SGLang.\n\nIssues with Unified Scheduling\nThe conventional unified engine, which processes prefill and decode batches together, results in two significant problems:\nPrefill Interruption: Incoming prefill batches frequently interrupt ongoing decode batches, causing substantial delays in token generation.\nDP Attention Imbalance: In data-parallel (DP) attention, one DP worker may process a prefill batch while another handles a decode batch simultaneously, leading to increased decode latency.\nPD Disaggregation resolves these by separating the two stages, enabling tailored optimizations for each.\nFor the design details, please refer to link.\nCurrently, we support Mooncake and NIXL as the transfer engine.\n\nProfiling in PD Disaggregation Mode\nWhen you need to profile prefill or decode workers in PD disaggregation mode, please refer to the Profile In PD Disaggregation Mode section in the Benchmark and Profiling guide. Due to torch profiler limitations, prefill and decode workers must be profiled separately using dedicated command-line options.\n\nRouter Integration\nFor deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the SGLang Model Gateway (former Router).\n\nMooncake\n\nRequirements\nuv pip install mooncake-transfer-engine\n\n\nIB Device Configuration\n--disaggregation-ib-device supports the following formats when using the Mooncake backend:\nShared device list for all GPUs: mlx5_0 or mlx5_0,mlx5_1\nPer-GPU JSON mapping: {\"0\": \"mlx5_0,mlx5_1\", \"1\": \"mlx5_2,mlx5_3\"}\nPath to a JSON file containing the same per-GPU mapping\nEach JSON value uses the same comma-separated device list format as the shared configuration.\n\nUsage\n\nLlama Single Node\npython -m sglang.launch_server \\\n --model-path meta-llama/Llama-3.1-8B-Instruct \\\n --disaggregation-mode prefill \\\n --port 30000 \\\n --disaggregation-ib-device mlx5_roce0\npython -m sglang.launch_server \\\n --model-path meta-llama/Llama-3.1-8B-Instruct \\\n --disaggregation-mode decode \\\n --port 30001 \\\n --base-gpu-id 1 \\\n --disaggregation-ib-device mlx5_roce0\npython -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000\n\n\nDeepSeek Multi-Node\n# prefill 0\npython -m sglang.launch_server \\\n --model-path deepseek-ai/DeepSeek-V3-0324 \\\n --disaggregation-ib-device ${device_name} \\\n --disaggregation-mode prefill \\\n --host ${local_ip} \\\n --port 3"