"Skip to content\nNavigation Menu\nPlatform\nSolutions\nResources\nOpen Source\nEnterprise\nPricing\nSign in\nSign up\nvllm-project\n/\nvllm\nPublic\n Sponsor\nNotifications\nFork 19.4k\n Star 86.3k\nCode\nIssues\n2k\nPull requests\n3.8k\nDiscussions\nActions\nProjects\nSecurity and quality\n54\nInsights\n[RFC]: Heterogeneous TP<->DP parallelism across the P/D disaggregation boundary (MoRIIO)\n #46107\nNew issue\nOpen\nDescription\nedwinlim0919\nopened \nLast edited by edwinlim0919\nContributor\n[RFC]: Heterogeneous TP↔DP parallelism across the P/D disaggregation boundary (MoRIIO)\nMotivation\n\nDisaggregated prefill/decode lets the two phases run with different optimal parallelism, but MoRIIO today assumes the prefill and decode engines share the same parallelism layout (DP8EP↔DP8EP, or TP8↔TP8).\n\nThe two phases can benefit from different parallelism configurations. For example, on an 8K-in/1K-out DeepSeek-R1 workload (run on MI300X / gfx942):\n\nPrefill is latency-bound → TP wins TTFT (TP8 ≈ 1s vs DP8EP ≈ 10s).\nDecode is throughput-bound at higher concurrency → DP/EP scales (DP8EP keeps scaling ≥ c256).\n\nSo at high concurrencies, the best deployment option could be heterogeneous. With prefill nodes configured in TP8 mode and decode nodes configured in DP8EP mode, we can achieve low TTFT from the TP8 prefill, and scalable throughput at high request loads (e.g. 512 concurrent request load).\n\nThe performance results below show the benefit of the heterogeneous setup. Various concurrency levels were tested, which control the maximum number of requests that are allowed to being served at once in the P/D setup. For each concurrency level, 10 * C total requests were sent, where C is the concurrency. Warmups were also done to prevent cold start from affecting the reported performance.\n\nAt the highest concurrency tested (512 concurrency requests), the mixed parallelism 2P_TP8:1D_DP8EP setup achieves the highest throughput and lowest TTFT of the three configurations.\n\nConfig 1: 2P_TP8 : 1D_TP8 (homogeneous TP8)\nConcurrency\tTotal throughput (tok/s)\tGeneration throughput (tok/s)\tMean TPOT (ms)\tMean TTFT (ms)\tBenchmark duration (s)\nc32\t8,603\t961\t30.3\t1,270\t307.8\nc128\t27,576\t3,051\t37.6\t2,497\t385.6\nc512\t38,407\t4,263\t55.9\t55,179\t1,106.2\nConfig 2: 2P_DP8EP : 1D_DP8EP (homogeneous DP8EP)\nConcurrency\tTotal throughput (tok/s)\tGeneration throughput (tok/s)\tMean TPOT (ms)\tMean TTFT (ms)\tBenchmark duration (s)\nc32\t6,893\t770\t34.9\t4,448\t384.2\nc128\t22,749\t2,517\t37.9\t10,123\t467.4\nc512\t34,935\t3,878\t42.3\t77,155\t1,216.1\nConfig 3: 2P_TP8 : 1D_DP8EP (MIXED)\nConcurrency\tTotal throughput (tok/s)\tGeneration throughput (tok/s)\tMean TPOT (ms)\tMean TTFT (ms)\tBenchmark duration (s)\nc32\t7,291\t814\t36.2\t1,362\t363.2\nc128\t21,760\t2,408\t48.7\t2,535\t488.6\nc512\t42,349\t4,701\t55.0\t46,052\t1,003.2\n\nOne target topologies has been enabled and validated by the PRs in this RFC:\n\nForward: P_DP8EP → D_TP8 (DP/EP prefill, TP decode). This is the configuration described above.\n\nTODO in future work: We did not do extensive validation on the mirror version, since this configuration doesnt have good performance characteristics in our evaluations. Future work can validate it, but it is considered out of scope of this RFC.\n\nMirror : P_TP8 → D_DP8EP (TP prefill, DP/EP decode). The mirror of the configuration described above.\nProposed Change\n\nThe P/D KV transfer must route reads across engines whose (dp_size, tp_size) differ. Design (describing behaviour, not the diff):\n\nSource-rank resolution lives in the connector, not the proxy.\nThe proxy only does coarse instance + DP-rank request distribution; the KV source rank is a connector decision because only the connector knows the KV layout (MLA-replicated vs partitioned).\n\nPer-(dp, tp) session keying.\nSessions/handshake state are keyed by (engine, dp_rank, tp_rank) (get_engine_name_with_dp_tp) so each remote prefill rank gets its own session + memory-region metadata, instead of a single DP-only key.\n\nFlexible prefill-TP read (this RFC's headline mechanism).\nFor a TP-replicated prefill (prefill TP + MLA) feeding a TP1 decode, the latent KV is replicated across all prefill TP ranks, so any rank is a valid source. Previously, heterogeneous parallelism PD setups would only read from a single TP rank (TP0), rather than distributing READ load across multiple ranks. The connector picks a prefill TP rank per read via deterministic round-robin to spread reads and prefill NIC load across TP0...N-1. Gated on: world_size==1 (decode TP1 → no local TP collective to keep in lockstep over the choice), use_mla (KV replicated), and remote_dp_size==1 && remote_tp_size>1 (a pure-TP prefill with >1 rank). Outside the gate the source rank stays fixed by the local-rank mapping. This is byte-identical to current behaviour.\nFuture work:\nThe gate currently requires a pure-TP prefill (remote_dp_size == 1). An intra-node DP×TP prefill (e.g. DP4TP2) falls back to the fixed-rank read; extending the random spread to \"a random TP rank within the request's DP partition\" is a localized follow-up (the (dp, tp) threading is already in place).\n\nConsistent rank threading. The chosen rank flows through the session key, the build-on-demand handshake, the peer-host resolution and the read-completion notify port, so a read and its completion notify always address the same prefill rank (a drift leaks the prefill buffer → MR overflow).\n\nImplementation PRs\n\nThe implementation is split into focused, independent PRs, all standalone on\nmain (no cross-dependencies):\n\n[Core][KV-transfer] MoRIIO: heterogeneous TP<->DP prefill/decode read routing #46116 [Core][KV-transfer]: heterogeneous TP<->DP read routing: DP-rank read routing, per-(dp, tp) session keying, the flexible prefill-TP read (§3), and an eager pre-forward handshake so mixed configs don't deadlock.\n[Bugfix] MoRIIO toy P/D proxy: fix DP-rank index aliasing + harden for high-concurrency bursts #46115 [Bugfix]: toy-proxy DP-rank index-aliasing fix (the §1 request distribution layer) + high-concurrency proxy hardening.\n[Bugfix][KV-transfer] "