"Skip to content\nNavigation Menu\nPlatform\nSolutions\nResources\nOpen Source\nEnterprise\nPricing\nSign in\nSign up\nvllm-project\n/\nvllm\nPublic\n Sponsor\nNotifications\nFork 19.4k\n Star 86.3k\nCode\nIssues\n2k\nPull requests\n3.8k\nDiscussions\nActions\nProjects\nSecurity and quality\n54\nInsights\n[Feature]: [RFC] Bottleneck-aware expansion for PD disaggregation\n #43470\nNew issue\nOpen\nDescription\nzsn6493\nopened \nLast edited by zsn6493\n\nThis is an RFC-style feature request for discussion.\n\nI have been reading through the recent PD, KV transfer, and offloading work, and it feels like the next missing step is not replacing PD, but making PD deployments more adaptive when one side becomes the dominant bottleneck.\n\nSummary\n\nThis proposal does not replace PD disaggregation.\n\nInstead, it builds on the current PD model and asks whether vLLM should identify the dominant bottleneck at runtime and expand the most effective side first.\n\nConcretely, the idea is:\n\nkeep the current PD architecture\ndetect whether runtime pressure is mainly on prefill, decode, or decode-side KV capacity\nexpand the easiest and most effective side first\nreduce user-facing contention before relying on stronger competition inside a fixed topology\nMotivation\n\nPD already gives an important separation between prefill and decode pools, and KV transfer / request migration / offloading are all moving in the right direction.\n\nBut even with PD, deployments still seem mostly statically provisioned. That means it is still easy to end up in situations like:\n\nprefill is saturated while decode is relatively idle\ndecode is saturated while prefill is underutilized\ndecode-side KV capacity becomes the dominant bottleneck even when compute is still available\nremote KV transfer backlog creates hidden stalls between the two pools\n\nSo from my perspective, PD solves mixed-pool interference, but not yet dynamic imbalance between the pools.\n\nProposal\n\nIntroduce a bottleneck-aware expansion layer for PD disaggregation.\n\nA simple first version could classify runtime pressure into three categories:\n\nPrefill-bound\nPrefill-side compute or memory pressure dominates.\n\nDecode-bound\nDecode-side token throughput or scheduling pressure dominates.\n\nDecode-KV-bound\nDecode-side KV holding capacity dominates, even if decode compute is not yet saturated.\n\nThen the runtime (or a future policy layer built on top of it) could choose the most effective expansion action:\n\nPrefill-bound -> add prefill capacity\nDecode-bound -> add decode capacity\nDecode-KV-bound -> add decode-side KV capacity / tier\n\nThe key point is that this would make PD more adaptive without changing the core PD architecture.\n\nSuggested First Scope\n\nA practical first step might be:\n\nexpose structured runtime pressure signals\nclassify bottlenecks into Prefill-bound, Decode-bound, or Decode-KV-bound\nsupport one minimal expansion hook or policy target\n\nEven just that would already make PD more adaptive, without requiring full runtime elasticity.\n\nNon-Goals\n\nThis is not proposing:\n\nreplacing PD disaggregation\ngeneric runtime GPU hot-plug\nruntime TP/NCCL topology mutation within an active worker group\nfull cluster autoscaling inside vLLM\nRelated Work\n\nThis seems closely related to, but different from, the following threads:\n\nPD disaggregation roadmap: [Roadmap]: PD Disaggregation with NixlConnector Roadmap #33702\nKV transfer push-model discussion: [RFC]: [KV Connector]: Support KV push from Prefill to Decode node using Nixl Connector #36923\nKV cache offloading RFC: [RFC]: KV cache offloading #19854\nMulti-tier offloading RFC: [RFC]: Multi-tier KV offloading via the vLLM offloading connector #38260\nNIXL handshake / autoscaling decode discussion: [Bug]: nixl handshake is slow and can accumulate in a batch of requests where each request is pulling from a different prefill node #19777\n\nMy understanding is that those efforts already provide the right building blocks. What seems missing is a policy/runtime layer that decides which side should be expanded first when pressure emerges inside an already disaggregated deployment.\n\nQuestions\n\nI would really appreciate feedback on a few points:\n\ndoes this direction fit the current PD roadmap?\nwhich bottleneck class would be the best first target?\nshould a first step focus only on signaling, or include one minimal expansion hook?\nhow should this interact with ongoing KV transfer / request migration work?\nActivity\nzsn6493\nadded \nfeature request\nNew feature or request\n \nzsn6493\nchanged the title \n[-]\n[Feature]: [RFC] Bottleneck-aware elastic expansion for PD disaggregation\n[/-]\n \n[+]\n[Feature]: [RFC] Bottleneck-aware expansion for PD disaggregation\n[/+]\n \nzsn6493\nmentioned this \n[Roadmap]: PD Disaggregation with NixlConnector Roadmap #33702\nrobertgshaw2-redhat commented \nrobertgshaw2-redhat\nSponsor\nCollaborator\n\nI think you are spot on about the operational challenges!\n\nI think one challenge with implementing this IN VLLM is that the higher level orchestration system is really the place that makes the decision about whether the request should be processed with P/D or not. vLLM already enables running disagg and agg on the same model server\n\nin llm-d for instance, we could use the flow control https://llm-d.ai/docs/guides/flow-control feature to detect saturation of P or D pools, and then make scheduling decisions based on this\n\nim sure dynamo has an equivalent concept too\n\nIm happy to help facilitate introductions to either community if you want to experiment with these routing algorithms\n\n👍\nReact with 👍\n1\nzsn6493 commented \nzsn6493\nAuthor\n\n@robertgshaw2-redhat Thanks, this is very helpful.\n\nI think this boundary makes sense. My main concern is reducing runtime contention in a way that improves user experience, not necessarily forcing all elasticity logic into vLLM itself.\n\nUsing vLLM as the execution/runtime layer and letting the orchestration layer decide agg vs P/D based on saturation or flow-control signals feels like the right first experiment.\n\nThe llm-d flow control path sounds especi"