"Skip to content\nNavigation Menu\nPlatform\nSolutions\nResources\nOpen Source\nEnterprise\nPricing\nSign in\nSign up\nvllm-project\n/\nvllm\nPublic\n Sponsor\nNotifications\nFork 19.4k\n Star 86.3k\nCode\nIssues\n2k\nPull requests\n3.8k\nDiscussions\nActions\nProjects\nSecurity and quality\n54\nInsights\n[Performance]: decoding speed on long context\n #11286\nNew issue\nClosed as not planned\nDescription\n155394551lzk\nopened \nLast edited by 155394551lzk\nProposal to improve performance\n\nIn our experiments, we found that the decoding speed of vLLM decreases dramatically when the length of the prompt becomes longer.\nWe fixed the batchsize=90 the decoding speed is 5364 tokens/s when the length of the prompt is within 100, 5500 tokens/s when 100 to 200, and decreases to 782 when 4000 to 8000, and decreases to 273 when greater than 8000.\n\nprompt length\t0-100\t100-200\t200-500\t500-1000\t1000-2000\t2000-4000\t4000-8000\t8000+\nwords/s\t5364\t5500\t4722\t2815\t2484\t1627\t782\t273\nGPU is single A800, 80G, vLLM block_size=16, max_num_seqs=512, max_model_len=8192, max_tokens=200. Is that why page attention is accessed more often?\nReport of performance regression\n\nNo response\n\nMisc discussion on performance\n\nNo response\n\nYour current environment (if you think it is necessary)\nThe output of `python collect_env.py`\n\nBefore submitting a new issue...\nMake sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.\nActivity\nNext\n155394551lzk\nadded \nperformance\nPerformance-related issues\n \nnoooop commented \nnoooop\nLast edited by noooop\nCollaborator\n\nMost of the time, the GPU meet bandwidth bottleneck rather than computing bottleneck,\nSo the inference speed depends on the memory bandwidth.\n\nAs the prompt length increases, the size of the kv cache that needs to be read even exceeds the size of the model,\nThis is why inference speed is decrease.\n\nGo further:\n\nInference latency increases linearly to the context size, primarily due to the time needed to access\ncached tokens\n\nYou can even fit a linear function, x=kv cache size need read, y=time required for one step\n\nsee more flashdecoding\n\njoerunde\nmentioned this \n[Core] Reduce TTFT with concurrent partial prefills #10235\nFlynn-Zh commented \nFlynn-Zh\n\nGPU L40, Qwen2.5-32B-GPTQ-Int4\nsame question, prompt has 9k words, vllm takes 12 seconds, sglang 8 seconds.\nIs there any configuration that can improve performance?\n\nnoooop commented \nnoooop\nLast edited by noooop\nCollaborator\n\n@Flynn-Zh\n\nBy default\n\nvllm use flash_attn for decoding\nsglang use flashinfer for decoding\n\nflashinfer was slightly faster than flash_attn, but I'm not sure if that's still the case\n\nYou can try vllm + flashinfer see if that improve performance.\n\nLooking forward to your benchmark\n\nFlynn-Zh commented \nFlynn-Zh\nLast edited by Flynn-Zh\n\n@noooop\n\ni can't found the configuration to use flashinfer, how to use flashinfer in vllm?\n\nnoooop commented \nnoooop\nCollaborator\ninstall flashinfer\n\nI'm not sure vllm supports the latest released flashinfer v0.2.0 #11314\n\nIt is safer to use flashinfer v0.1.6\n\nset environment variable VLLM_ATTENTION_BACKEND=FLASHINFER， enable flashinfer\nFlynn-Zh commented \nFlynn-Zh\n\nuse flashinfer still 12 seconds\n\nnoooop commented \nnoooop\nCollaborator\n\nuse flashinfer still 12 seconds\n\ninteresting\n\njeejeelee commented \njeejeelee\nMember\n\nMaybe it is a similar issue with #11317 (comment)\n\nFlynn-Zh commented \nFlynn-Zh\n\n@jeejeelee i try increase max-seq-len-to-capture，but it's useless\n\nnoooop commented \nnoooop\nCollaborator\n\n@Flynn-Zh\n\nvllm v0 use default scheduler，chunked_prefill performs better for long inputs\n\nPlease try the configuration below:\n\nenable_chunked_prefill = True\nmax_num_seqs=32\nmax_num_batched_tokens=2048 <- 2048 token can generally make the GPU reach saturation\n\nFlynn-Zh commented \nFlynn-Zh\n\n@noooop\nI've also tried it，it's useless\n\njeejeelee commented \njeejeelee\nLast edited by jeejeelee\nMember\n\n@jeejeelee i try increase max-seq-len"