"Skip to main content\narXiv is now an independent nonprofit!\nLearn more\n×\nSearch\nSubmit\nDonate\nLog in\nComputer Science > Distributed, Parallel, and Cluster Computing\n[Submitted on 24 Jun 2024 (v1), last revised 3 Sep 2025 (this version, v4)]\nMooncake: A KVCache-centric Disaggregated Architecture for LLM Serving\nRuoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu\nMooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake's innovative architecture enables Kimi to handle 75% more requests.\nComments:\t23 pages, 13 figures\nSubjects:\tDistributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR)\nCite as:\tarXiv:2407.00079 [cs.DC]\n \t(or arXiv:2407.00079v4 [cs.DC] for this version)\n \t\nhttps://doi.org/10.48550/arXiv.2407.00079\nFocus to learn more\nSubmission history\nFrom: Ruoyu Qin [view email]\n[v1] Mon, 24 Jun 2024 02:05:32 UTC (264 KB)\n[v2] Tue, 2 Jul 2024 02:49:35 UTC (264 KB)\n[v3] Tue, 9 Jul 2024 04:03:10 UTC (280 KB)\n[v4] Wed, 3 Sep 2025 14:56:29 UTC (245 KB)\n\nAccess Paper:\nView PDF\nHTML (experimental)\nTeX Source\nview license\nCurrent browse context:\ncs.DC\n< prev | next >\n\nnew | recent | 2024-07\nChange to browse by:\ncs\ncs.AI\ncs.AR\n\nReferences & Citations\nNASA ADS\nGoogle Scholar\nSemantic Scholar\nExport BibTeX Citation\nBookmark\n \nBibliographic Tools\nBibliographic and Citation Tools\nBibliographic Explorer Toggle\nBibliographic Explorer (What is the Explorer?)\nConnected Papers Toggle\nConnected Papers (What is Connected Papers?)\nLitmaps Toggle\nLitmaps (What is Litmaps?)\nscite.ai Toggle\nscite Smart Citations (What are Smart Citations?)\nCode, Data, Media\nDemos\nRelated Papers\nAbout arXivLabs\nWhich authors of this paper are endorsers? | Disable MathJax (What is MathJax?)\nWe gratefully acknowledge support from our major funders, member institutions, and all contributors.\nAbout\n·\nHelp\n·\nContact\n·\nSubscribe\n·\nCopyright\n·\nPrivacy\n·\nAccessibility\n·\nOperational Status\n(opens in new tab)\nMajor funding support from"