"Bỏ qua để đến phần nội dung chính\nHỗ trợ truy cập\nĐăng nhập\nChế độ AI\nTất cả\nVideo ngắn\nMua sắm\nHình ảnh\nVideo\nTin tức\nThêm\nCông cụ\nKết quả tìm kiếm\nLooking for results in English?\nChange to English\nTiếp tục sử dụng Tiếng Việt\nCài đặt ngôn ngữ\nThông tin tổng quan do AI tạo\nThe information you are referring to comes from the June 2026 arXiv paper titled \"Demystifying the Design Space and Best Practices for Heterogeneous Disaggregated Serving\". The paper systematically analyzes how heterogeneous infrastructure impacts Prefill-Decode (PD) disaggregation, particularly focusing on the friction caused by differing Key-Value (KV) cache formats and precision policies across different hardware. \narXiv\n +1\nBelow is an organized breakdown of the design space axes, core boundaries, and best practices outlined in the research.\nThe Four Axes of the Heterogeneous PD Design Space\nThe paper structures the heterogeneous disaggregated serving landscape along four primary system axes: \narXiv\nAccelerator Architecture: Mixing different vendor hardware (e.g., NVIDIA, AMD, or specialized ASICs) across the prefill and decode stages. \narXiv\n +1\nPrecision Policy: Using separate quantization granularities or bit-widths (e.g., FP16, INT8, FP4) for weights, activations, and KV caches. \narXiv\n +3\nInterconnect Topology: The physical network layer linking the stages, ranging from ultra-fast PCIe/NVLink to standard Ethernet or CPU-assisted fallback paths. \narXiv\nKV Residency: The storage strategy managing where the KV cache sits during transfers (e.g., GPU VRAM, host system DRAM, or multi-tier storage). \narXiv\n +1\nThree Core Boundary Decisions & Friction Points\nWhen prefill and decode nodes use different hardware or software runtimes, three crucial \"boundary decisions\" emerge where independent optimizations conflict: \narXiv\n +1\n [ Prefill Stage ] [ Decode Stage ]\n (Optimizes Compute) (Optimizes Memory)\n\n | |\n +------> [ The Heterogeneous Boundary ] <-------+\n - Precision Alignment\n - KV Layout Compatibility\n - Hand-off & Lifecycle Ownership\n\n1. Compute Placement & Precision Policy\nThe Conflict: Prefill is compute-bound, whereas decode is memory-bandwidth bound. A low-bit quantization strategy optimized to relieve memory pressure on a decode node might lack an executable kernel path or hardware tensor core support on a heterogeneous prefill accelerator. \narXiv\n +2\nBest Practice: Precision policies must be treated as runtime-specific roles rather than system-wide global configurations. The prefill and decode engines should dynamically apply precision formats that solve their localized bottlenecks without forcing the opposing stage into an inefficient execution path. \narXiv\n2. KV Representation & Layout Compatibility\nThe Conflict: Different runtimes (such as vLLM or TensorRT-LLM) structure KV caches differently using distinct page sizes, memory layouts, padding, or attention variants. If a prefill node transmits an incompatible tensor structure, the decode engine must waste costly cycles restructuring it. \narXiv\n +3\nBest Practice: KV transfer engines must behave strictly as byte-movers rather than semantic tensor transfers. Layout compatibility must be treated as an explicit edge boundary concern, embedding metadata definitions natively into the transfer layer so that translation happens seamlessly during network transit. \narXiv\n3. KV Ownership and Lifecycle Management\nThe Conflict: Splitting the inference pipeline creates distributed state management problems. Runtimes can experience synchronization failures regarding memory reservation, cache release, and node failure recovery across the network fabric. \narXiv\n +1\nBest Practice: Implement a unified, cross-stage lifecycle protocol that explicitly assigns KV ownership. This guarantees predictable memory allocation, minimizes fragmentation, and ensures rapid cleanup or recomputation if a decode worker drops offline. \narXiv\n +1\nIf you would like to explore deeper, tell me if you are building on a homogenous or heterogeneous hardware cluster, or which LLM serving engine you are planning to optimize.\n6 trang web\nDemystifying the Design Space and Best Practices for ... - arXiv\n30 thg 6, 2026 — Most prior work studies these mechanisms independently. PD-disaggregated serving separates prompt processing from autoregressive g...\narXiv\n[2606.29708] Demystifying the Design Space and Best Practices for ...\n30 thg 6, 2026 — We propose a design space organized along four design axes-accelerator, precision, interconnect, and KV residency and the workload...\narXiv\nPrefill-as-a-Service: KVCache of Next-Generation Models ...\n16 thg 4, 2026 — Making this design work requires scheduling and cache management that explicitly address the remaining systems challenges. Conside...\narXiv\nHiện tất cả\nHiện thêm\nWeb results\nDemystifying the "