"Documentation Index\n\nFetch the complete documentation index at: /llms.txt\n\nUse this file to discover all available pages before exploring further.\n\nSkip to main content\nSGLang Documentation home page\nSearch...\n⌘K\nAsk Assistant\nGet Started\nUser Guide\nHardware\nCookbook\nSGLang Diffusion\nBasic Usage\nBasic Usage\nOpenAI-Compatible APIs\nAnthropic-Compatible API\nOllama-Compatible API\nOffline Engine API\nSGLang Native APIs\nSampling Parameters\nAdvanced Features\nAdvanced Features\nServer Arguments\nHyperparameter Tuning\nAttention Backend\nHiSparse: Hierarchical Sparse Attention\nSpeculative Decoding\nAdaptive Speculative Decoding\nStructured Outputs\nStructured Outputs For Reasoning Models\nTool Parser\nReasoning Parser\nQuantization\nQuantized KV Cache\nDP, DPA and SGLang DP Router\nExpert Parallelism\nLoRA Serving\nPD Disaggregation\nEPD Disaggregation\nPipeline Parallelism for Long Context\nHierarchical KV Caching (HiCache)\nQuery VLM with Offline Engine\nDP for Multi-Modal Encoder in SGLang\nCuda Graph for Multi-Modal Encoder in SGLang\nBreakable CUDA Graph\nPiecewise CUDA Graph\nSGLang Model Gateway\nllm-d\nDeterministic Inference\nObservability\nModel Loading\nLoading Models from Object Storage\nCheckpoint Engine Integration\nSGLang for RL Systems\nSupported Models\nSupported models\nText Generation\nRetrieval and Ranking\nSpecialized Models\nExtending SGLang\nDeveloper Guide\nDeveloper Guide\nContribution Guide\nDevelopment\nBenchmarking\nEvaluating New Models with SGLang\nMSProbe Debugging Guide\nReferences\nReferences\nTroubleshooting and Frequently Asked Questions\nEnvironment Variables\nProduction Metrics\nProduction Request Tracing\nDeployment\nCustom Chat Template\nFrontend Language\nCookbook\nPost-Training Integration\nNightly precision regression\nAdvanced Features\nPipeline Parallelism for Long Context\nCopy page\n\nWhy Pipeline Parallelism?\nAs Large Language Models (LLMs) scale toward trillion-parameter architectures and “infinite” context windows, the underlying serving infrastructure must evolve toward more granular, cross-node parallelization strategies. While KV cache techniques effectively mitigate redundant computation, they cannot circumvent the prohibitive Time to First Token (TTFT) inherent in ultra-long sequences with extremely large initial Input Token Length (ITL). Although Tensor Parallelism (TP) remains the conventional approach for intra-node scaling, it frequently encounters communication bottlenecks during multi-node deployments. On the other hand, pipeline parallelism only requires cross-node communication at the boundaries of each pipeline stage, which can achieve better computation-communication overlap compared to a large TP. Therefore, it is also a promising parallelization strategy for improving throughput.\nDetailed analysis can be found in this blog.\n\nImplementation Refactoring based on Async Communication\nWith Dynamic Chunked Prefill, pipeline parallelism has the potential to reduce the TTFT of long-context inputs. For each request, its input tokens can be partitioned into multiple chunks, each no longer than the chunked prefill size. Different chunks of the same request can be processed simultaneously by different nodes, thus parallelizing the processing and reducing TTFT. SGLang has supported Pipeline Parallelism (#5724) for some time and made it compatible with the PD Disaggregation feature (#8846), but the implementation was not perfect and had significant room for performance improvements.\nTo eliminate this performance hazard, SGLang implements a Micro-batching Event Loop with non-blocking asynchronous peer-to-peer (P2P) communication to overlap GPU computation with CPU metadata processing and PP communication. This ensures that while one micro-batch is being computed on the GPU, the next one is already being prepared and moved into position effectively, ensuring the pipeline remains as saturated as possible. This approach was first proposed in #7979 and has been redesigned and included in #11852.\nThe key mechanisms of the implementation include:\nDecoupled Sync/Async Logic in the Event Loop: The scheduler uses async_send in _pp_send_pyobj_to_next_stage. Instead of waiting for a transfer to complete, it returns a P2PWork handle. The actual synchronization (P2PWork.work.wait()) is deferred until _pp_commit_comm_work is called, allowing the CPU to perform other work—like scheduling the next batch or processing metadata—while data is in flight.\nMulti-Stream Execution: In addition to the main default_stream, which serves as the synchronization stream, SGLang utilizes dedicated forward_stream and copy_stream to execute forward pass GPU computation and Data-to-Host (D2H) memory transfers separately for better overlapping. While _pp_launch_batch is executing the current micro-batch on the GPU for the current stage, the CPU processes the previous micro-batch’s results using _pp_process_batch_result.\n\nGuidance about Dynamic Chunking\n\nWhy Dynamic Chunking\nChunked prefill with a fixed size can cause bubbles in the pipeline, especially when the pp size is large. The main reason behind this phenomenon is that the model has a non-uniform running time, even though each chunk size is identical (brought by the Transformer structure). The larger the prefix sequence length, the longer the running time of the chunk. And these bubbles will be propagated to the next stage, and will significantly degrade the scale efficiency of larger pp ranks.\nTo address this issue, SGLang introduces a dynamic chunking mechanism to predict the optimal size for the next chunk such that it satisfies this condition:\nRuntime(L + Next Chunk Size) - Runtime(L) = Runtime(Initial Chunk Size)\nwhere L denotes the Prefix Sequence Length. By profiling a series of requests with different ITLs, we model the cumulative runtime as a quadratic function of sequence length. Using this model, we solve the optimal next chunk size for any given prefix length L. Since the computation complexity of the Attention mechanism scales with L, the next chunk size will be progres"