"Skip to content\nNavigation Menu\nPlatform\nSolutions\nResources\nOpen Source\nEnterprise\nPricing\nSign in\nSign up\nsgl-project\n/\nsglang\nPublic\nNotifications\nFork 7.2k\n Star 30.3k\nCode\nIssues\n706\nPull requests\n3.6k\nDiscussions\nActions\nSecurity and quality\nInsights\nPD disaggregation decode deadlocks at high per-rank batch size with CUDA graphs (all ranks spinning in flashinfer moeA2ACombineKernel)\n #30010\nNew issue\nOpen\nDescription\nyhyang201\nopened \nCollaborator\nEnvironment\nsglang main a7390b17f (2026-07-01), flashinfer-python 0.6.12, driver 580.159.03\nGB300 nodes (4 GPUs/node, MNNVL fabric), Kimi-K2.5-NVFP4, mooncake transfer backend\nSymptom\n\nUnder sustained decode load with per-DP-rank running requests above ~450, the decode worker wedges within 4–8 minutes (6/6 runs across EP4 and EP8 decode shapes):\n\nAll scheduler ranks stuck in result.copy_done.synchronize() at scheduler_components/batch_result_processor.py:635 (py-spy).\nGPUs at 100% util: cuda-gdb shows most ranks spinning in tensorrt_llm::kernels::moe_alltoall::moeA2ACombineKernel<__nv_bfloat16, ..., 8> on the same grid, while one rank is behind (caught once in moe::dev::finalize::finalizeKernelVecLoad, two grids earlier) and later shows an empty GPU.\nThe lagging rank eventually surfaces torch.AcceleratorError: CUDA error: unspecified launch failure; the watchdog then kills the worker (~300 s after the last decode batch log).\nIsolation matrix (all measured on the same model/config)\nScenario\tCUDA graphs\tPer-rank occupancy\tResult\nPD disagg serving (mooncake)\ton\t450–790\tdeadlock 6/6, 4–8 min\nPD disagg serving\t--disable-cuda-graph\t512 (slot cap)\tstable 26 min\nSingle-node agg, fixed-bs decode loop (bench_one_batch, bs 2048 × 4 rounds)\ton\t512\tstable\nSingle-node agg serving, dynamic load\ton\t730–790\tstable 16 min\n\nThe failure needs PD disaggregation + CUDA graphs + high per-rank batch size simultaneously. Excluded: streaming settings (--stream-interval 1/8/50 all reproduce), EP domain size (EP4 and EP8 both die; EP8 dies at lower occupancy), client shape, bucket churn alone (agg dynamic load is fine). Not caused by message corruption — reproduced with zero errors after we separately fixed an unrelated sender-side ZMQ race in the mooncake path.\n\nReproduction (verified 3/3)\n\n3 nodes: decode TP8 spans node0+node1, prefill on node2, router on node2. (Also reproduces with a single-node TP4/EP4/DP4 decode worker at running >~500/rank.)\n\nDecode, node0 (--node-rank 1 on node1, same command otherwise):\n\nMC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1 \\\nSGLANG_MOONCAKE_CUSTOM_MEM_POOL=True \\\nSGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK=4096 \\\npython3 -m sglang.launch_server \\\n --model-path nvidia/Kimi-K2.5-NVFP4 --trust-remote-code \\\n --tp 8 --ep 8 --dp-size 8 --enable-dp-attention --enable-dp-lm-head \\\n --nnodes 2 --node-rank 0 --dist-init-addr <NODE0_IP>:20000 \\\n --quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 \\\n --mem-fraction-static 0.85 --context-length 9216 --disable-radix-cache \\\n --moe-a2a-backend flashinfer --moe-runner-backend flashinfer_trtllm_routed \\\n --attention-backend trtllm_mla --disable-flashinfer-autotune \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --max-running-requests 4096 --cuda-graph-max-bs 2048 \\\n --tokenizer-worker-num 8 --detokenizer-worker-num 4 --stream-interval 50 \\\n --host 0.0.0.0 --port 30000\n\nPrefill, node2:\n\nMC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1 \\\nSGLANG_MOONCAKE_CUSTOM_MEM_POOL=True \\\nSGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32768 \\\npython3 -m sglang.launch_server \\\n --model-path nvidia/Kimi-K2.5-NVFP4 --trust-remote-code \\\n --tp 4 --ep 4 --dp-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 \\\n --mem-fraction-static 0.77 --context-length 9216 --disable-radix-cache \\\n --moe-a2a-backend flashinfer --moe-runner-backend flashinfer_trtllm_routed \\\n --attention-backend trtllm_mla --disable-flashinfer-autotune \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --chunked-prefill-size 32768 --max-prefill-tokens 32768 --max-running-requests 512 \\\n --tokenizer-worker-num 4 --host 0.0.0.0 --port 30000\n\nRouter, node2:\n\npython3 -m sglang_router.launch_router --pd-disaggregation \\\n --prefill http://<NODE2_IP>:30000 --decode http://<NODE0_IP>:30000 \\\n --host 0.0.0.0 --port 30080\n\nBench:\n\npython3 -m sglang.bench_serving --backend sglang-oai --host <NODE2_IP> --port 30080 \\\n --dataset-name random --random-input-len 1024 --random-output-len 1024 \\\n --random-range-ratio 0.8 --num-prompts 24576 --max-concurrency 8192 --request-rate inf\n\nThis prefill config sustains ~76 req/s, which pushes decode past ~450 running/rank within ~2 minutes; the wedge follows within minutes. With the default --chunked-prefill-size 16384 prefill, supply is lower, decode stays under ~400/rank, and the system runs stable for tens of minutes — which is why this is easy to miss.\n\nSuspicions\n\nPD-only components that interact with graph replay: (a) KV cache allocated from the mooncake custom mem pool (SGLANG_MOONCAKE_CUSTOM_MEM_POOL, cuMemCreate/fabric memory) vs the plain allocator in agg mode; (b) concurrent mooncake NVLink writes into the KV pool while captured graphs replay; (c) the disagg-only prebuilt-batch forward path. Also note the eager dispatcher has a runtime_max_tokens_per_rank exceeds max_num_tokens assert with no equivalent protection inside captured graphs. Possibly related: flashinfer-ai/flashinfer#3279 (P0, NVFP4 MoE EP8 nondeterministic hang introduced in 0.6.8.post1, driver-version sensitive).\n\nWorkarounds\nCap decode admission so per-rank running stays ≤ ~400 (e.g. --max-running-requests 3200 for DP8).\n--disable-cuda-graph (stable but slow).\n\nFull forensics (py-spy stacks of all ranks, cuda-gdb kernel dumps, timelines) available on request.\n\nActivity\nadityakamat24 commented \nadityakamat24\nContributor\n\nHey @yhyang201 , can I look into this? Thanks!\n\nSign up for fr"