"Projects\nBlog\nAbout\nDonations\nContact\n‹ Back to Blog\n\nCONTENTS\n\nTL;DR\nIntroduction\nThe ViT Scaling Problem: Why Tensor Parallelism Doesn't Always Help\nThe Counter-Intuitive Finding\nArchitecture Overview\nKey Components\nImplementation Details\nImage Distribution Strategies\nTransfer Backends\nVision Embedding Cache\nUsage Examples\nBenchmark\nExperimental Setup\nBench Results\nAcknowledgment\nLearn more\nEPD Disaggregation: Elastic Encoder Scaling for Vision-Language Models in SGLang\nrednote hilab, Alibaba Cloud Computing, AntGroup SCT\nJanuary 12, 2026\nTL;DR\n\nWe introduce Encoder-Prefill-Decode (EPD) Disaggregation in SGLang, a novel architecture that separates vision encoding from language processing in Vision-Language Models (VLMs). This can enable:\n\nIndependent scaling of vision encoding capacity: Encoder servers can be scaled horizontally without affecting language model deployment, enabling better resource utilization for vision-heavy workloads.\nCompatibility with existing PD disaggregation: EPD can be combined with Prefill-Decode disaggregation for a complete three-tier architecture.\nFlexible transfer backends: Support for multiple transfer mechanisms (ZMQ, GPU-direct via Mooncake) allows optimization for different deployment scenarios.\nVision embedding caching: Frequently used images can be cached at encoder servers, eliminating redundant ViT computations and reducing network transfer overhead.\n\nEPD is highly effective in image-heavy scenarios (e.g., multi-image inputs), where the visual encoding process is the primary computational bottleneck. For instance, in these scenarios, we leverage EPD to significantly reduce request TTFT under load—achieving approximately 6–8× lower latency compared to the colocation approach at 1 QPS. Conversely, for image-light scenarios with few images, EPD may be less efficient or even counterproductive. This is because the additional network latency incurred by transmitting embeddings across nodes can outweigh the time saved by offloading the encoding task, potentially resulting in a higher TTFT compared to a colocation approach.\n\nIntroduction\n\nVision-Language Models (VLMs) like Qwen2.5-VL and Llama-Vision combine visual understanding with language generation. However, these models face unique scaling challenges:\n\nHeterogeneous compute needs: Vision encoding (CNN/ViT) requires different computational patterns than language decoding (Transformer)\nImbalanced resource usage: Vision processing is compute-intensive but only needed during prefill\nLimited flexibility: Traditional monolithic deployments can't independently scale vision and language components\nIntra-request parallelism: Different images in one request can be encoded independently.\nPoor scaling under tensor parallelism: Because the vision encoder has far fewer parameters than the language component, applying tensor parallelism to it is inefficient and generally unnecessary.\n\nSGLang's existing Prefill-Decode (PD) disaggregation already separates prefill from decode phases. EPD extends this by further separating vision encoding from language prefill, creating a three-tier architecture.\n\nThe ViT Scaling Problem: Why Tensor Parallelism Doesn't Always Help\nThe Counter-Intuitive Finding\n\nOne of the key insights from EPD disaggregation is that Vision Transformers (ViT) do NOT benefit from increased Tensor Parallelism, and can actually become slower with higher TP:\n\nBenchmark on H20 with Qwen2.5-VL-72B (4 images per request):\n\ntp\tvit mean time\n2\t492.13ms\n4\t465.80ms\n8\t523.80ms\n\nWhy Does This Happen?\n\nCommunication overhead dominates execution time.\nThe weight parameters of vision models are usually small.\n\nEPD sidesteps this by scaling encoders horizontally instead of increasing TP.\n\nArchitecture Overview\n\nThe EPD architecture follows a request flow:\n\nClient Request: A multimodal request arrives at the prefill server (via load balancer or direct connection).\nImage Distribution: The prefill server identifies image inputs and distributes them to one or more encoder servers. Images can be split across multiple encoders for load balancing.\nVision Encoding: Encoder servers process images through ViT, generating vision embeddings and image grid metadata. Results are cached if enabled.\nEmbedding Transfer: Vision embeddings are transferred back to the prefill server using the configured transfer backend (ZMQ, Mooncake, etc.).\nLLM Computation: The prefill server combines vision embeddings with text tokens to form mm_inputs containing pre-computed tensors. The LLM performs Prefill and Decode computation based on these embeddings. If PD disaggregation is enabled, the existing Prefill-Decode transfer logic is reused; otherwise, token generation happens directly on the prefill server.\nKey Components\n\nEncoder Server (--encoder-only)\n\nVision-only (no language weights); preprocessing + ViT forward to generate vision embeddings\nPrefix multimodal cache support\nScale out for load balancing and parallel multi-image split inference\n\nPrefill Server (--language-only)\n\nLanguage model only\nReceives embeddings from encoder(s)\nWith PD: ships KV to Decode; without PD: decodes locally\n\nDecode Server\n\nStandard decode-only instance\nReceives KV cache from prefill\nImplementation Details\nImage Distribution Strategies\n\nUnlike tensor parallelism which splits a single model across GPUs, EPD uses data parallelism by running multiple independent encoder instances and distributing images among them.\n\nExample distribution:\n\nRequest with 7 images: [img0, img1, img2, img3, img4, img5, img6]\n3 encoders available\n\nDistribution (after shuffle):\n├─ Encoder 0: [img0, img1, img2] (3 images)\n├─ Encoder 1: [img3, img4] (2 images)\n└─ Encoder 2: [img5, img6] (2 images)\n\nTransfer Backends\n\nEPD supports three transfer backends for vision embeddings:\n\nzmq_to_scheduler (Default)\n\nDirect ZMQ socket communication\nEmbeddings sent from encoder to scheduler via RDMA transfer engine.\nNo blocking\n\nzmq_to_tokenizer\n\nEmbeddings sent to tokenizer manager\nProcessed during tokeniza"