What You Will Find Here
Hand-written analysis covering AI system design, chip architecture, and infrastructure economics.
My observations, analysis, and insights — personal perspectives on AI infrastructure, chips, and system design.
BaM claims 4.4×~21.8× cost advantage over DRAM-only systems. But this conclusion rests on $/GB — while the real bottleneck for these workloads is IOPS. When IOPS demand scales beyond the tested 10-SSD configuration, the cost curve may bend in ways the paper never measured.
CXL/RDMA distributed memory pools upgrade memory from compute appendage to poolable network resource, but existing data analytics tooling (pandas/cuDF/Ray/Arrow) still assumes memory is a local compute-node appendage. This architectural mismatch demands redesign at every layer — from UCX communication to Arrow zero-copy protocol to DataFrame memory models.
Extend PyTorch Dataset/DataLoader API to build CXL memory pool-aware distributed data loading system — not a transparent solution, but explicitly leveraging CXL semantics (byte-level addressing, NUMA-aware, tiered placement) for global data pooling, intelligent prefetching, and load balancing across hundreds of TB of memory.
The critical missing piece in the CXL-for-training stack is not hardware capability, but a fundamental rethinking of PyTorch's DataLoader architecture. This report provides source-level analysis of PyTorch DataLoader internals, identifies exact integration points for CXL, and systematically answers five critical design questions — from data path to architectural philosophy.
GPU-Driven RDMA is not just 'GPU initiates RDMA' — it is the tight coupling of memory and communication in a multi-execution-unit system (GPU + NIC + CPU), analogous to the multi-core era where cache coherence and memory ordering became first-class design problems. This article explores the five fundamental challenges this convergence creates: data transfer, ordering, consistency, network parallelism, and memory flexibility.
OpenURMA is an open-source FPGA implementation of the UB (Ultra-Bus) connectionless RDMA transport, achieving ≈500ns remote load/store latency on Alveo U50 (vs 2236ns on RoCE, 4.47× faster), while fully implementing the UB-Base-Specification 2.0.1 transaction and transport layers.
Re-examining DeepSeek's MoE communication stack across five layers — from the physical network fabric up to global algorithmic scheduling — reveals that 'lock-free' and 'fully overlapped compute-communication' never actually eliminate synchronization. They redistribute it across physical-link queuing, microarchitectural bus arbitration, hardware-level memory-consistency fences, and algorithmic data-dependency waits.
Ant Group's HSA-UltraLong demonstrates 16M token context via Hierarchical Sparse Attention. But sparsity dilutes during prefill — when sequence length grows 10×, a new storage tier with TB capacity and 500GB–1TB bandwidth becomes mandatory. This post analyzes the infra implications.
DeepEP began as an EP acceleration library for MoE models. Its rebranding to "Every Parallelization" signals a structural ambition: becoming the thin data-transfer abstraction layer that unifies operators, QoS, and direct-drive capabilities across rack- and cluster-scale compute. This post traces that trajectory, draws the NCCL parallel, and argues why a new unified memory-data-movement service layer is emerging.
A showcase of AI-generated short videos — stunning fidelity, visceral impact, almost indistinguishable from reality. First installment of a multimodal AI survey series.
DeepSeek-V4-Flash-0731 vs Kimi-K3 benchmark comparison. Kimi leads on all 5 shared benchmarks, but the two models target different segments — Flash efficiency vs flagship all-rounder.
以文言译英文原文，论多符预测（MTP）如何移推理之负载从访存密集转向计算密集，及其对芯片、互连、超节点之结构性冲击。
DeepSeek's MTP doesn't just optimize inference — it restructures the compute-memory-interconnect triangle. This post analyzes the cascading impact on chip design, communication paradigms, supernode topology, and the competitive landscape of specialized silicon, with hard data from NVIDIA LPX, Groq Trillium, Cerebras WSE-3, and the linear-vs-sparse architecture war.
DeepEP is not an All-to-All communication library — it is a Data Movement Runtime for MoE. Its core challenge is converting dynamic sparse Token-Expert mappings into continuous GPU-friendly data layouts. From first principles, this article analyzes DeepEP's complete design: layout transformation, Buffer pipeline, Warp Specialization, FIFO streaming, Metadata mapping, and the evolution toward Mega MoE.
Warp Specialization transforms GPU from a batch-synchronous SIMT machine into a streaming dataflow engine. By assigning different Warps to different pipeline stages, it sacrifices local SIMD efficiency for global pipeline efficiency — the key to overlapping communication and compute in MoE dispatch, FlashAttention, and beyond.
GPU's traditional execution model assumes Thread Blocks are independent — but modern AI workloads break this assumption. Hopper's two key evolutions — raising the Resident Block limit and introducing Thread Block Cluster — represent a fundamental shift: from optimizing single-Tensor-Core utilization to optimizing on-chip dataflow across the entire GPU.
Unified addressing is often discussed as a programming-ease feature. But the real reason we need an independent, global address service is deeper: it is about amortizing the cost of establishing communication pipes, enabling data services at every latency tier to exchange data with lower overhead. From Redis to NVLink, the history of AI distributed systems is a history of address services evolving toward lower latency.
Unified addressing is not a fixed implementation — it is a design philosophy. Location Transparency can be established at any scope and at any time. The current NVL72 global fabric is one deliberate trade-off among many. Future AI supernodes need a dynamic Address Resolution Service and an Object Directory.
Claude Code runs tools locally — but products from DeepSeek, Manus, Kimi, Volcano Engine, Meituan, and E2B are building server-side execution into their core architecture. This post analyzes real product designs to explain why server-side CPU demand is growing.
The frontier model landscape is splitting into two directions: cost-efficiency (DeepSeek) and high-precision (Kimi3, Claude, GPT). For AI infrastructure, this bifurcation is not a problem — it's a design constraint that defines the storage hierarchy.
CXL is not just another interconnect protocol. From a market game theory lens, I believe CXL's long-term positioning is structurally underappreciated. Four forces are converging to make CXL the inevitable backbone of open-memory infrastructure.
Google's KV Cache strategy isn't one solution — it's two complementary approaches: node-local tiered storage (HBM + CPU RAM + Local SSD) for single-node efficiency, and centralized Lustre for multi-tenant sharing. Both prove the same point: storage is the bottleneck.
A faithful analysis of Kimi3's efficiency claims from the Linear paper, with cost modeling at million-token scale and the fundamental tradeoffs between linear and sparse attention architectures.
Kimi3 represents the frontier of linear attention architectures — but can it scale? This post analyzes the micro-architecture war between linear attention (Kimi3, Qwen3) and sparse attention (DeepSeek V4, GLM52, LongCat), with quantitative analysis of their scaling ceilings.
I ran an Agent session that accumulated 480 million tokens of context history. The bill revealed a truth that changes how we should think about AI infrastructure: at scale, storage costs dominate compute costs by orders of magnitude.