sglang_23022_shm_pd

"Skip to content\n\nYou are viewing the latest developer preview docs. Click here to view docs for the latest stable release.\n\nvLLM\nDisaggregated Prefilling (experimental)\n \nInitializing search\n GitHub\nv0.25.1\n86.2k\n19.4k\nvLLM\nHome\nUser Guide\nUser Guide\nGetting Started\nGetting Started\nQuickstart\nInstallation\nInstallation\nGPU\nCPU\nTPU\nExamples\nExamples\nApplications\nApplications\nAPI Server\nChatbot\nRag\nBasic\nBasic\nOffline Inference\nOnline Serving\nDeployment\nDeployment\nAsync LLM Streaming\nHelm Charts\nLLM Engine Example\nSagemaker-Entrypoint\nDisaggregated\nDisaggregated\nDisaggregated Encoder\nDisaggregated Serving\nEc Both Encoder\nDisaggregated Prefill V1\nFlexkv Connector\nKV Load Failure Recovery Test\nLMCache Examples\nMooncake Connector\nFeatures\nFeatures\nAutomatic Prefix Caching\nBatch Invariance\nContext Extension\nData Parallel\nKv Events\nLogging Configuration\nCustom Logits Processors\nLoRA\nOffline Inference with the OpenAI Batch file format\nPause Resume\nProfiling\nPrompt Embed\nReset Kv\nSharded State\nSpeculative Decoding\nStructured Outputs\nTensorize vLLM Model\nTorchrun\nGenerate\nGenerate\nBatched Chat Completions Online\nMultimodal\nQwen 1M Offline\nObservability\nObservability\nMonitoring Dashboards\nMetrics\nSetup OpenTelemetry POC\nPrometheus and Grafana\nPooling\nPooling\nClassify\nEmbed\nPlugin\nReward\nScore\nToken Classify\nToken Embed\nRay Serving\nRay Serving\nBatch LLM Inference\nElastic Ep\nMulti-Node-Serving\nRay Serve Deepseek\nRun Cluster\nReasoning\nReasoning\nOpenAI Chat Completion Tool Calls With Reasoning\nOpenAI Chat Completion With Reasoning\nOpenAI Chat Completion With Reasoning Streaming\nOpenAI Responses Client\nRL\nRL\nRLHF Async New APIs\nRLHF Http IPC\nRLHF Http NCCL\nRLHF IPC\nRLHF IPC Fsdp Ep\nRLHF NCCL\nRLHF NCCL Fsdp Ep\nRLHF Sparse NCCL\nRouted Experts E2E\nSkip Loading Weights In Engine Init\nScale Out\nScale Out\nInit\nExample Mm Serve\nToken Generation Client\nSpeech To Text\nSpeech To Text\nLid\nOpenAI\nRealtime\nTool Calling\nTool Calling\nChat With Tools Offline\nOpenAI Chat Completion Client With Tools\nOpenAI Chat Completion Client With Tools Required\nOpenAI Chat Completion Client With Tools Xlam\nOpenAI Chat Completion Client With Tools Xlam Streaming\nOpenAI Responses Client With Mcp Tools\nOpenAI Responses Client With Tools\nGeneral\nGeneral\nvLLM V1\nFrequently Asked Questions\nProduction Metrics\nReproducibility\nSecurity\nTroubleshooting\nUsage Stats Collection\nInference and Serving\nInference and Serving\nOffline Inference\nOnline Serving\nOnline Serving\nGenerative Scoring\nOpenAI-Compatible Server\nRenderer APIs\nSpeech to Text APIs\nContext Parallel Deployment\nData Parallel Deployment\nTroubleshooting distributed deployments\nExpert Parallel Deployment\nParallelism and Scaling\nIntegrations\nIntegrations\nClaude Code\nCodex\nLangChain\nLlamaIndex\nDeployment\nDeployment\nUsing Docker\nUsing Kubernetes\nUsing Nginx\nFrameworks\nFrameworks\nAnyscale\nAnythingLLM\nAutoGen\nBentoML\nCerebrium\nChatbox\nDify\ndstack\nHaystack\nHelm\nHugging Face Inference Endpoints\nLiteLLM\nLobe Chat\nLWS\nModal\nOpen WebUI\nRetrieval-Augmented Generation\nRunPod\nSkyPilot\nStreamlit\nNVIDIA Triton\nIntegrations\nIntegrations\nAIBrix\nNVIDIA Dynamo\nKAITO\nKServe\nKthena\nKubeAI\nKubeRay\nLlama Stack\nllm-d\nllmaz\nProduction stack\nTraining\nTraining\nAsync Reinforcement Learning\nWhat is Layerwise (Re)loading?\nReinforcement Learning from Human Feedback\nTransformers Reinforcement Learning\nWeight Transfer\nWeight Transfer\nBase Class and Custom Engines\nIPC Engine\nNCCL Engine\nConfiguration\nConfiguration\nConserving Memory\nEngine Arguments\nEnvironment Variables\nModel Resolution\nOptimization and Tuning\nServer Arguments\nTPU\nModels\nModels\nSupported Models\nGenerative Models\nPooling Models\nPooling Models\nClassification Usages\nEmbedding Usages\nReward Usages\nScoring Usages\nSpecific Model Examples\nToken Classification Usages\nToken Embedding Usages\nExtensions\nExtensions\nLoading model weights with fastsafetensors\nLoading Model Weights with InstantTensor\nLoading models with Run:ai Model Streamer\nLoading models with CoreWeave's Tensorizer\nHardware Supported Models\nHardware Supported Models\nCPU - Intel® Xeon®\nXPU - Intel® GPUs\nTPU\nFeatures\nFeatures\nAutomatic Prefix Caching\nBatch Invariance\nContext Extension\nCustom Arguments\nCustom Logits Processors\nDisaggregated Encoder\nDisaggregated Prefilling (experimental)\nIndexCache\nInterleaved Thinking\nKV Offloading Usage Guide\nLoRA Adapters\nMooncakeConnector Usage Guide\nMooncakeStoreConnector Usage Guide\nMoRIIOConnector Usage Guide\nMultimodal Inputs\nNixlConnector Compatibility Matrix\nNixlConnector Usage Guide\nPer-Request Metrics\nPrompt Embedding Inputs\nReasoning Outputs\nSleep Mode\nStructured Outputs\nTool Calling\nQuantization\nQuantization\nAutoAWQ\nBitsAndBytes\nFP8 ViT Encoder Attention\nGGUF\nGPTQModel\nIntel Quantization Support\nNVIDIA Model Optimizer\nOnline Quantization\nQuantized KV Cache\nAMD Quark\nTorchAO\nLLM Compressor\nLLM Compressor\nFP8 W8A8\nINT4 W4A16\nINT8 W4A8\nINT8 W8A8\nSpeculative Decoding\nSpeculative Decoding\nDraft Models\nDynamic Speculative Decoding\nEAGLE Draft Models\nHidden State Extraction\nMLP Draft Models\nMTP (Multi-Token Prediction)\nN-Gram Speculation\nParallel Draft Models\nvLLM-Project/Speculators\nSuffix Decoding\nDeveloper Guide\nDeveloper Guide\nGeneral\nGeneral\nDeprecation Policy\nDockerfile\nEditing Agent Instructions\nIncremental Compilation Workflow\nProfiling vLLM\nVulnerability Management\nModel Implementation\nModel Implementation\nBasic Model\nRegistering a Model\nUnit Testing\nMulti-Modal Support\nSpeech-to-Text (Transcription/Translation) Support\nCI\nCI\nCI Failures\nNightly Builds of vLLM Wheels\nUpdate PyTorch version on vLLM OSS CI/CD\nDesign Documents\nDesign Documents\nPlugins\nPlugins\nEndpoint Plugins\nIO Processor Plugins\nLoRA Resolver Plugins\nPlugin System\nArchitecture Overview\nAttention Backend Feature Support\nCUDA Graphs\nVision Encoder (ViT) CUDA Graphs\nCustomOp\nDual Batch Overlap\nHow to debug the vLLM-torch.compile integration\nFused MoE Modular Kernel\nFusion torch.compile passes\nIntegration with Hugging Face\nHybrid KV Cache Manager\nLogits Processors\nMetrics\nMulti-Modal Data Proce"