"Skip to main content\narXiv is now an independent nonprofit!\nLearn more\n×\nSearch\nSubmit\nDonate\nLog in\nComputer Science > Distributed, Parallel, and Cluster Computing\n[Submitted on 16 Apr 2026 (v1), last revised 22 Apr 2026 (this version, v2)]\nPrefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter\nRuoyu Qin, Weiran He, Yaoyu Wang, Zheming Li, Xinran Xu, Yongwei Wu, Weimin Zheng, Mingxing Zhang\nPrefill-decode (PD) disaggregation has become the standard architecture for large-scale LLM serving, but in practice its deployment boundary is still determined by KVCache transfer. In conventional dense-attention models, prefill generates huge KVCache traffics that keep prefill and decode tightly coupled within a single high-bandwidth network domain, limiting heterogeneous deployment and resource elasticity. Recent hybrid-attention architectures substantially reduce KVCache size, making cross-cluster KVCache transport increasingly plausible. However, smaller KVCache alone does not make heterogeneous cross-datacenter PD serving practical: real workloads remain bursty, request lengths are highly skewed, prefix caches are unevenly distributed, and inter-cluster bandwidth fluctuates. A naive design that fully externalizes prefill can therefore still suffer from congestion, unstable queueing, and poor utilization.\nWe present Prefill-as-a-Service (PrfaaS), a cross-datacenter serving architecture that selectively offloads long-context prefill to standalone, compute-dense prefill clusters and transfers the resulting KVCache over commodity Ethernet to local PD clusters for decode. Rather than treating reduced KVCache as sufficient, PrfaaS combines model-side KV efficiency with system-side selective offloading, bandwidth-aware scheduling, and cache-aware request placement. This design removes the requirement that heterogeneous accelerators share the same low-latency RDMA fabric, enabling independent scaling of prefill and decode capacity across loosely coupled clusters. In a case study using an internal 1T-parameter hybrid model, a PrfaaS-augmented heterogeneous deployment achieves 54% higher serving throughput and 64% lower P90 TTFT than a homogeneous PD baseline, with approximately 15% throughput gain at equal cost, while consuming only modest cross-datacenter bandwidth.\nComments:\t16 pages, 5 figures, 6 tables\nSubjects:\tDistributed, Parallel, and Cluster Computing (cs.DC)\nCite as:\tarXiv:2604.15039 [cs.DC]\n \t(or arXiv:2604.15039v2 [cs.DC] for this version)\n \t\nhttps://doi.org/10.48550/arXiv.2604.15039\nFocus to learn more\nSubmission history\nFrom: Ruoyu Qin [view email]\n[v1] Thu, 16 Apr 2026 14:07:41 UTC (244 KB)\n[v2] Wed, 22 Apr 2026 03:05:12 UTC (244 KB)\n\nAccess Paper:\nView PDF\nHTML (experimental)\nTeX Source\nview license\nCurrent browse context:\ncs.DC\n< prev | next >\n\nnew | recent | 2026-04\nChange to browse by:\ncs\n\nReferences & Citations\nNASA ADS\nGoogle Scholar\nSemantic Scholar\nExport BibTeX Citation\nBookmark\n \nBibliographic Tools\nBibliographic and Citation Tools\nBibliographic Explorer Toggle\nBibliographic Explorer (What is the Explorer?)\nConnected Papers Toggle\nConnected Papers (What is Connected Papers?)\nLitmaps Toggle\nLitmaps (What is Litmaps?)\nscite.ai Toggle\nscite Smart Citations (What are Smart Citations?)\nCode, Data, Media\nDemos\nRelated Papers\nAbout arXivLabs\nWhich authors of this paper are endorsers? | Disable MathJax (What is MathJax?)\nWe gratefully acknowledge support from our major funders, member institutions, and all contributors.\nAbout\n·\nHelp\n·\nContact\n·\nSubscribe\n·\nCopyright\n·\nPrivacy\n·\nAccessibility\n·\nOperational Status\n(opens in new tab)\nMajor funding support from"