"Skip to main content\narXiv is now an independent nonprofit!\nLearn more\n×\nSearch\nSubmit\nDonate\nLog in\nComputer Science > Distributed, Parallel, and Cluster Computing\n[Submitted on 2 Jul 2026]\nLynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference\nWenchen Han, Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste, Adam Barker\nLong-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems. In disaggregated inference, these workloads require transferring large Key-Value (KV) caches across the network, where decoding cannot begin until the transfer completes. Recent KV quantization techniques reduce data volume and alleviate this bottleneck, but existing schemes fail to achieve both low network-exposed latency and high inference accuracy.\nWe challenge the assumption that the KV cache is an indivisible unit that must be fully received before use. We leverage the observation that different bits in the KV cache contribute unequally to attention computation and inference precision: the most significant bits capture the coarse structure of attention and the least significant bits refine precision. This property enables partial use of the KV cache during decoding. We present Lynx, a system that enables progressive, split-stream KV transfer by partitioning the KV cache into a high-priority Anchor stream carrying the most significant bits and a low-priority Residual stream carrying remaining precision. Decoding begins upon receipt of the Anchor stream and proceeds speculatively while the Residual stream is transferred concurrently, followed by verification that ensures equivalence to higher-precision decoding.\nAcross multiple models and serving workloads, Lynx achieves Time-to-First-Token (TTFT) comparable to aggressive 4-bit KV quantization, while matching the accuracy of high-precision (BF16) inference, improving TTFT over standard 8-bit KV quantization by up to \n1.43×\n and improving accuracy over state-of-the-art by up to \n5.1%\n.\nComments:\t15 pages, 12 figures. This manuscript was originally submitted to SIGCOMM '26 in February 2026\nSubjects:\tDistributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)\nACM classes:\tC.2.4; I.2.11\nCite as:\tarXiv:2607.01831 [cs.DC]\n \t(or arXiv:2607.01831v1 [cs.DC] for this version)\n \t\nhttps://doi.org/10.48550/arXiv.2607.01831\nFocus to learn more\nSubmission history\nFrom: Wenchen Han [view email]\n[v1] Thu, 2 Jul 2026 07:52:43 UTC (1,756 KB)\n\nAccess Paper:\nView PDF\nTeX Source\nview license\nCurrent browse context:\ncs.DC\n< prev | next >\n\nnew | recent | 2026-07\nChange to browse by:\ncs\ncs.LG\n\nReferences & Citations\nNASA ADS\nGoogle Scholar\nSemantic Scholar\nExport BibTeX Citation\nBookmark\n \nBibliographic Tools\nBibliographic and Citation Tools\nBibliographic Explorer Toggle\nBibliographic Explorer (What is the Explorer?)\nConnected Papers Toggle\nConnected Papers (What is Connected Papers?)\nLitmaps Toggle\nLitmaps (What is Litmaps?)\nscite.ai Toggle\nscite Smart Citations (What are Smart Citations?)\nCode, Data, Media\nDemos\nRelated Papers\nAbout arXivLabs\nWhich authors of this paper are endorsers? | Disable MathJax (What is MathJax?)\nWe gratefully acknowledge support from our major funders, member institutions, and all contributors.\nAbout\n·\nHelp\n·\nContact\n·\nSubscribe\n·\nCopyright\n·\nPrivacy\n·\nAccessibility\n·\nOperational Status\n(opens in new tab)\nMajor funding support from"