"Skip to content\nNavigation Menu\nPlatform\nSolutions\nResources\nOpen Source\nEnterprise\nPricing\nSign in\nSign up\nsgl-project\n/\nsglang\nPublic\nNotifications\nFork 7.2k\n Star 30.3k\nCode\nIssues\n706\nPull requests\n3.6k\nDiscussions\nActions\nSecurity and quality\nInsights\n[Bug] DeepSeek-V4-Pro，PD Disaggregation Input Length Hard-Capped at 18,432 Tokens Due to Incorrect SWA Limit Enforcement on Prefill Node\n #24523\nNew issue\nOpen\nDescription\ntxh1873749380\nopened \nLast edited by txh1873749380\nChecklist\nI searched related issues but found no solution.\nThe bug persists in the latest version.\nIssues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.\nIf this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.\nPlease use English. Otherwise, it will be closed.\nDescribe the bug\n\nIn the PD-disaggregated deployment using the ds-v4-hopper image with the deepseek-v4-pro model and SGLang's official FP8 configuration, the following code path is triggered:\n\nif self.scheduler.tp_worker.is_hybrid_swa:\n # FIXME: current SWA allocation allocate full kv cache size in prefill\n self.max_total_num_tokens = min(\n self.max_total_num_tokens,\n self.scheduler.tp_worker.model_runner.swa_max_total_num_tokens,\n )\n\n[2026-05-06 09:15:41 DP7 TP7 EP7] Memory profiling: available_gpu_memory=17.94 GB, total_gpu_memory=138.17 GB, mem_fraction_static=0.90, rest_memory=4.13 GB\n[2026-05-06 09:15:41 DP7 TP7 EP7] DSv4 memory calculation: bytes_per_full_token=23749.84, available_bytes=4.13 GB, full_token=186368\n[2026-05-06 09:15:41 DP7 TP7 EP7] DSv4 pool sizes: full=186368, swa=18432, c4=46592, c128=1456, c4_state=1152, c128_state=18432\n[2026-05-06 09:15:41 DP6 TP6 EP6] Memory profiling: available_gpu_memory=17.94 GB, total_gpu_memory=138.17 GB, mem_fraction_static=0.90, rest_memory=4.13 GB\n[2026-05-06 09:15:41 DP6 TP6 EP6] DSv4 memory calculation: bytes_per_full_token=23749.84, available_bytes=4.13 GB, full_token=186368\n[2026-05-06 09:15:41 DP5 TP5 EP5] Memory profiling: available_gpu_memory=17.94 GB, total_gpu_memory=138.17 GB, mem_fraction_static=0.90, rest_memory=4.13 GB\n[2026-05-06 09:15:41 DP6 TP6 EP6] DSv4 pool sizes: full=186368, swa=18432, c4=46592, c128=1456, c4_state=1152, c128_state=18432\n[2026-05-06 09:15:41 DP5 TP5 EP5] DSv4 memory calculation: bytes_per_full_token=23749.84, available_bytes=4.13 GB, full_token=186368\n\n\n python3 -c \"\nimport json\ntext = ('hell word。' + '\\n') * 4800\npayload = {\n 'model': '/data/models/DeepSeek-V4-Pro',\n 'messages': [{'role': 'user', 'content': text}],\n 'max_tokens': 400,\n 'temperature': 0.7\n}\nwith open('/tmp/80k_req.json', 'w') as f:\n json.dump(payload, f)\nprint('chars:', len(text), 'estimated tokens:', int(len(text) * 84006 / 136000))\n\" && curl -s http://172.16.0.8:8889/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d @/tmp/80k_req.json\nchars: 163200 estimated tokens: 100807\n{\"error\":{\"type\":\"Bad Request\",\"code\":\"decode_bad_request\",\"message\":\"Request 212ff413b9d844ed9d944c70e9ab6c1e exceeds the maximum number of tokens: 100804 > 18432\"}}\n\nBecause the prefill node operates under is_hybrid_swa = true, its effective token capacity is clamped to swa_max_total_num_tokens (18,432) instead of the full KV-cache budget (186,368). The FIXME comment explicitly notes that the current SWA allocator reserves the full KV cache size during prefill, yet the runtime still enforces the SWA upper bound on max_total_num_tokens. This is why the prefill node rejects any request exceeding 18,432 tokens, even though the hardware has sufficient HBM for the full 186k context window. @fzyzcjy @boxiaowave\n\nReproduction\n\nThe deployment runs on H200 GPUs, using the deepseek-v4-hopper container image and the deepseek-v4-fp8 model, following the PD disaggregation commands from the official documentation.\n\nEnvironment\n\nGPU：H200\nmodels：DeepSeek-V4-Pro\n\nActivity\ntxh1873749380\nchanged the title \n[-]\n[Bug] PD Disaggregation Input Length Hard-Capped at 18,432 Tokens Due to Incorrect SWA Limit Enforcement on Prefill Node\n[/-]\n \n[+]\n[Bug] DeepSeek-V4-Pro-FP8，PD Disaggregation Input Length Hard-Capped at 18,432 Tokens Due to Incorrect SWA Limit Enforcement on Prefill Node\n[/+]\n \ntxh1873749380 commented \ntxh1873749380\nAuthor\n\n@janimo\n\nCloudRipple commented \nCloudRipple\nContributor\n\nI'm encountering the same issue, but I'm using a prefill-decode (PD) disaggregated deployment, so the available context is a bit longer, with swa=91904. However, it remains severely unusable.\n\ntxh1873749380\nchanged the title \n[-]\n[Bug] DeepSeek-V4-Pro-FP8，PD Disaggregation Input Length Hard-Capped at 18,432 Tokens Due to Incorrect SWA Limit Enforcement on Prefill Node\n[/-]\n \n[+]\n[Bug] DeepSeek-V4-Pro，PD Disaggregation Input Length Hard-Capped at 18,432 Tokens Due to Incorrect SWA Limit Enforcement on Prefill Node\n[/+]\n \nShangmingCai commented \nShangmingCai\nCollaborator\n\nCC: @ispobock @weireweire\n\nweireweire commented \nweireweire\nContributor\n\nAre you using chunked prefill? I think SWA allocation allocates full KV cache size means it is not tail-only, but it may still be a chunk instead of the full prompt, and some old swa chunk can be evicted. In that case, comparing the full prompt length with swa_max_total_num_tokens may be overly conservative.\n\nWe should\nprobably compare swa_max_total_num_tokens against the estimated peak SWA requirement, e.g. chunk size plus sliding-window/page/overlap margin, while still checking the full prompt against the full KV capacity.\n\nCloudRipple commented \nCloudRipple\nContributor\n\nAre you using chunked prefill? I think SWA allocation allocates full KV cache size means it is not tail-only, but it may still be a chunk instead of the full prompt, and some old swa chunk can be evicted. In that case, comparing the full prompt length with swa_max_total_num_tokens may be overly conservative.\n\nWe should probably compare swa_max_total_n"