"Back to arXiv\n\nThis is experimental HTML to improve accessibility. We invite you to report rendering errors. \nUse Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off.\nLearn more about this project and help improve conversions.\n\nWhy HTML?\nReport Issue\nBack to Abstract\nDownload PDF\n1Introduction\n2Operational Intensity and Capacity Footprint\n3The heterogeneity panacea\n4Where Agentic AI inference systems are headed: Hypotheses\n5Conclusion\n References\n\nHTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.\n\nfailed: mdframed.sty\nfailed: mdframed.sty\n\nAuthors: achieve the best HTML results from your LaTeX submissions by following these best practices.\n\nLicense: CC BY-NC-SA 4.0\narXiv:2601.22001v1 [cs.AI] 29 Jan 2026\nHeterogeneous Computing: The Key to Powering the Future of AI Agent Inference\nAaron Zhao\nImperial College London a.zhao@imperial.ac.uk\n&Junyi Liu Microsoft Research junyi.liu@microsoft.com\n\n1Introduction\n\nThe rise of AI agents has significantly reshaped the landscape of computer systems. There are gigawatt-scale data centers to be built in the coming years, primarily to support AI workloads (Somala and Cottier, 2025). We are moving toward an inference-heavy future – reports have shown that AI agents are expected to power many existing and future applications, and running inference (just the forward pass) for these agents will likely become the dominating workloads in future AI data centers (McKinsey & Company, 2025; Morgan Stanley, 2025).\n\nHardware heterogeneity has become a key part of today’s cloud computing. Starting from optimizing CPU-oriented workloads, we have seen the adoption of networking accelerators like SmartNICs and parallel compute units like GPUs. These accelerators greatly improve both system performance and efficiency by offloading IO- and compute-intensive tasks. Since the release of ChatGPT, AI workloads have been predominantly optimized for the GPU-centric infrastructure. Today, a predominant belief is that more compute specialization will be needed for AI agent inference, and we see this trend has started in the inference chips like Nvidia’s Rubin CPX (NVIDIA, 2025). In fact, hardware heterogeneity is further spreading beyond the compute accelerators. Scalable and efficient inference for large-scale AI agents requires wider and spread-out heterogeneity at the system level ‐ across compute, networking, and memory.\n\nAI agent inference, when deployed across different scenarios, can now exhibit substantial variation in compute to memory bandwidth (FLOPs/bandwidth) and compute to memory capacity (FLOPs/capacity) ratios. This results from a combination of different agentic stack use cases (e.g., computer use, chatbot), base model architectures (e.g., different attentions employed, dense or sparse architectures), and distinct system and model-level optimizations (e.g., quantization, prefill-decode disaggregation). To satisfy such workload diversity, future scaling imposes unprecedented pressures on accelerators, interconnects, and memory systems. We argue that system‑level heterogeneity will, in turn, require cohesive integration at the datacenter scale.\n\n2Operational Intensity and Capacity Footprint\n\nWe define two key metrics – related to the capacity and bandwidth requirements on the memory side – to better characterize the inference process of a given AI agent.:\n\n• \n\nOperational Intensity (OI): The number of operations performed per byte of data moved from DRAM, this is a classic metric widely used in the roofline models.\n\n• \n\nCapacity Footprint (CF): The number of bytes needed per agent request in DRAM for LLM generation. The product of batch size and CF states the capacity requirement on the DRAM.\n\nConsider a simple matrix multiplication \n𝐘\n=\n𝐖𝐗\n, wher"