§执行摘要
Sparse Clos 并非单一论文提出的确切术语，而是网络拓扑设计领域对一类"降低 Clos 互联密度"改进方案的统称——通过减少/取消 spine 层、引入过订阅（oversubscription）、采用部分 mesh / 准随机互联或可重构光交换（OCS）等手段，在成本/功耗/线缆复杂度与吞吐/弹性/可扩展性之间重新平衡。在 AI 大模型训练（尤其 MoE all-to-all 通信）驱动万卡/十万卡集群的当下，Sparse Clos 已从学术构想走向 hyperscaler 生产现实。
核心结论（5 点）
1953Clos 网络起源（Bell Labs）
1:2.8Meta Llama4 跨区过订阅
69%AWS RNG 较 fat-tree 少路由器
41%Google Jupiter OCS 降功耗
$200M30k GPU 全阻塞 Clos 成本
- 历史脉络清晰：Clos(1953) → fat-tree/spine-leaf(2000s) → Jellyfish 随机图(2012) → Dragonfly/SlimFly 低直径(2008/2016) → OCS 直连稀疏化(2022 Google Jupiter) → 准随机扁平拓扑(2026 AWS RNG)。
- "稀疏化"有两条技术路线：① 结构稀疏（减 spine、部分 mesh、过订阅）；② 动态稀疏（OCS/光交换按需重构拓扑）。Google 走②，AWS/Meta 走①与②的混合。
- ECMP 是 Clos 性能天花板：静态哈希在 all-to-all 下只能利用 40-80% 容量，hash 冲突可使二分带宽下降 60.8%。稀疏化后必须配套动态负载均衡（Hedera/DRB/DRILL/RailS）。
- AI 负载改变设计前提：MoE all-to-all 占迭代时间 40-60%，流量稀疏不均衡，NIC/上行负载差可达 5-10×，倒逼 Rail-Only / 直连拓扑绕开 spine。
- 无银弹：拓扑设计是"平衡与妥协的工程艺术"，小规模 Zcube、中规模 Fat-tree、大规模 Zettafly 各有最优区间，不存在全尺度最优解。
1Clos 网络基础理论：1953 起源与数学性质
1.1 Clos 网络的诞生（1953）
1953 年，Bell 实验室的 Charles Clos 在 Bell System Technical Journal 发表论文，提出一种多级电路交换网络，用小规模交叉点矩阵组合出大规模无阻塞交换结构。其核心是三级结构 C(n, m, r)：r 个输入级模块（每个 n×m）、m 个中间级模块（每个 r×r）、r 个输出级模块（每个 m×n）。Clos 证明当 m ≥ 2n−1 时网络严格无阻塞（strictly non-blocking），当 m ≥ n 时可重排无阻塞（rearrangeably non-blocking）。
💡 概念锚点 Clos 的精髓在于：用"多级 + 并行路径"替代"单个巨型高 radix 交换"，以低成本小芯片堆叠出任意大规模无阻塞网络。这一思想 60 年后成为数据中心 spine-leaf 与 fat-tree 的理论根基。
1.2 现代数据中心的 Clos 变体：spine-leaf 与 fat-tree
当代数据中心将 Clos 理论工程化，形成两大主流结构，二者本质上都是 Clos：
| 拓扑 | Clos 关系 | 结构 | 无阻塞性 |
|---|
| Spine-Leaf（2 级 Clos） | 2 级 Clos | Leaf 与 Spine 全 mesh 互联；无固定参数约束层数/端口数，设计自由度高 | 可重排无阻塞（1:1 时） |
| Fat-Tree（折叠 Clos） | 5 级 Clos = 3 层 fat-tree | k 端口交换机构建 k 个 pod，(k/2)² 个 core 交换机；源-目的间有 (k/2)² 条等价路径（但非链路不相交） | 全二分带宽（1:1 时） |
leaf and spine, whose main characteristic is the establishment of two layers of switches ... no parameter governs the number of devices or ports among layers, so there is some freedom of design ... there is full mesh connectivity between switches located in both layers.
for a fat-tree network built from k-port switches, there are k pods ... The k pods are interconnected by (k/2)² core switches ... between any given source and destination host pair, there are (k/2)² equal-cost paths ... Note, however, that these paths are not link-disjoint.
等价性证据：多篇文献明确将 5 级 Clos 等同于 3 层 fat-tree：
Finding flow routes ... is NP-complete for integer flows. And while simultaneous flow routing is solvable in polynomial time for 3-stage Clos networks, no polynomial time algorithm is known for 5-stage Clos networks (i.e. 3-tier fat-trees).
We also use established three-stage fat trees (FT3) that are a variant of the Clos network.
1.3 Clos 为何"统治"数据中心
Clos is the most commonly deployed topology in data centers and supercomputers today, and it dominates the landscape of Ethernet clusters.
Clos 的优势：① 用廉价小端口 commodity 交换机堆出大规模网络；② 路径多样性——每对源目的间有多条并行路径；③ 模块化、可重复。Google 多代数据中心均采用 Clos：
Clos topology - a non-blocking multistage switching topology, built out of smaller radix switch chips, that can scale to arbitrarily large networks.
1.4 Clos 的固有缺陷（稀疏化的动因）
Clos 并非完美，其结构性缺陷正是 Sparse Clos 兴起的根本动因：
| 缺陷 | 表现 | 来源 |
|---|
| Spine 必须 all-to-all fanout | spine 须以最快设备速率运行；增量升级 spine 需重布整个数据中心线缆 | Google Jupiter |
| 设计粒度粗糙 | fat-tree 只能在 3456/8192/27648/65536 等离散规模构建（对应 24/32/48/64 端口）；增量扩容需替换全部交换机 | Jellyfish NSDI'12 |
| ECMP 静态哈希低效 | hash 冲突致二分带宽降 60.8%（all-to-all）；仅利用 40-80% 容量 | Hedera / DRB |
| 成本/功耗爆炸 | 30k GPU 全阻塞 Clos fabric $200M；10 万加速器 fat-tree 需数十亿美元 | Next Platform / 综述 |
Clos topologies require a spine layer with uniform support for the fastest devices that might connect to it ... This is because Clos topologies inherently require all-to-all fanout from aggregation blocks to the spine; adding to the spine incrementally would require rewiring the entire data center.
full bisection bandwidth fat-trees can only be built at sizes 3456, 8192, 27648, and 65536 corresponding to ... port counts of 24, 32, 48, and 64 ... this 'incremental' growth would require replacing all the 48-port switches by 50-port switches.
2Sparse Clos 设计原理：稀疏化定义与机制
"稀疏 Clos" 并无一篇奠基论文给出严格定义，而是文献中对降低 Clos 互联密度一类设计的描述性统称。其核心问题是：能否用比全阻塞（1:1）Clos 更少的互联资源，达到接近甚至超越的性能？
2.1 稀疏化的形式化表述
最直接给出"sparse"概念的来源是 TROD（光纤数据中心阈值路由-拓扑协同设计，IEEE/ACM ToN 2023）：
the ToR-level logical topology is much more sparse. One possible [question is] if it is possible for optical DCNs to attain better performance than the 1:1 Clos.
稀疏化的关键参数是 oversubscription ratio（过订阅比）：当上行总带宽 < 下行总带宽时，网络即"稀疏化"。Alibaba 的生产实验直接演示了这一点：
To evaluate the performance of C4P within a congested network environment, we intentionally reduced the number of active spine switches by half, effectively increasing the network oversubscription rate to 2:1.
✅ 稀疏化的最直接定义：在 leaf-spine Clos 中，减少 spine 交换机数量 → 过订阅比从 1:1 升至 2:1（或更高）。这是成本（少买 spine/光模块/线缆）与吞吐（二分带宽下降）的直接权衡。
2.2 四种稀疏化机制
| 机制 | 做法 | 代表 | 代价 |
|---|
| ① 减/删 spine | 直接减少 spine 数量或用片上交换替代 spine 层 | NVIDIA Rail-Only；Meta 1:2.8 过订阅 | 跨 rail 流量拥塞 |
| ② 部分 mesh / 准随机互联 | 用随机/准随机图替代结构化全 mesh | AWS RNG；Jellyfish | 需自适应路由 |
| ③ OCS 动态重构 | 光交换按需建立/拆除链路，逻辑拓扑随流量变 | Google Jupiter；OpenOptics；Meta DSF | 重构时延、控制复杂 |
| ④ 过订阅分层 | 不同层用不同过订阅比，跨层稀疏 | Meta Llama4 RTSW/CTSW/ATSW | 跨层集合通信限速 |
2.3 机制①：删除 spine（Rail-Only 与 Spineless）
HotNets 2020 的 "Spineless data centers" 直接挑战 spine 层的必要性：
over-subscription is 2x less in flat networks than leaf-spines.
more efficient topologies indeed exist at moderate scale, showing ... much of the benefit comes from choosing a 'flat' network that uses one type of switch rather than having separate roles for leafs and spines ... even a simple ring-based topology outperforms leaf-spine for a wide range of traffic scenarios.
工业界最激进的"删 spine"实践是 NVIDIA Rail-Only（Hot Interconnects 2024）：在 128 台 DGX H100（1024 GPU）集群中，用 HGX 板上的 NVSwitch ASIC 替代 spine 聚合层，配套数据极其可观：
20→8交换机数量
2688→1152光模块数量
41 kW省功耗
$1.3M省网络成本
With the rail-only network, you drop to eight switches for the rails and a mere 1,152 transceivers ... This will save 41 kilowatts of power and eliminate $1.3 million in network costs.
关键洞察：LLM 训练中绝大多数流量留在 rail 内（leaf 交换机），极少上达 spine：
Most of the traffic stays within the rails, and does not cut across the rails... Whether there is pipeline parallelism, tensor parallelism, or data parallelism, only very rarely does the traffic go up into those expensive spine switches.
2.4 机制③：OCS 动态稀疏（Google 范式）
Google Jupiter Evolving（SIGCOMM 2022）是动态稀疏化的旗舰案例：用 MEMS 光交换机（OCS）替代电气 spine，消除 spine 层，在聚合块间构建按需直连 mesh：
With an OCS layer, we eliminated the spine layer from our data center networks, instead connecting heterogeneous aggregation blocks in a direct mesh, for the first time moving beyond Clos topologies in the data center.
A key enabler ... is evolving Jupiter from a Clos to a direct-connect topology among the machine aggregation blocks. Critical architectural changes ... A datacenter interconnection layer employing Micro-ElectroMechanical Systems (MEMS) based Optical Circuit Switches (OCSes) to enable dynamic topology reconfiguration.
"稀疏"的量化：Jupiter 直连 fabric 中 60% 流量走直达路径（路径长 1），其余 40% 多跳一块，平均块级路径长度仅 1.4——并非全 mesh：
60% of the traffic takes direct path from source to destination aggregation blocks, while the remaining transits one additional block, achieving an average block-level path length of 1.4 in our fleet today.
2.5 机制②：准随机扁平（AWS RNG）
AWS 2026 年公开的 RNG（Resilient Network Graphs）是结构稀疏 + 随机化的代表，已成为大多数新建 AWS 数据中心的默认拓扑：
The resulting network design - which we call RNG, for resilient network graphs - is now used in AWS data centers and is the default for most new builds globally. It uses 69% fewer routers, delivers up to 33% better throughput, and projects a 40% reduction in network equipment electricity consumption.
RNG 的硬件创新是 ShuffleBox（无源光器件，内部走线打乱），使准随机布线的物理复杂度降到与 fat-tree 相当：
Our central innovation is a passive optical device called a ShuffleBox. ... The internal wires are shuffled according to a special pattern, so that random connections between the ShuffleBoxes lead to an overall quasi-random topology. When a new rack arrives, a technician plugs its router into an available port on the local ShuffleBox. No rewiring elsewhere.
3关键学术论文谱系：路由与负载均衡
稀疏 Clos 之所以可行，关键在于配套动态路由/负载均衡算法弥补结构稀疏带来的路径不平衡。以下构成一条清晰的学术谱系：
2010 · Hedera (NSDI)集中式动态流调度，8192 主机达 96% 最优二分带宽，较 ECMP 提升 113%。揭示 ECMP 在 Clos 多路径下的 hash 冲突本质。
2013 · DRB (CoNEXT)Per-packet Digit-Reversal Bouncing 路由，Clos 上达 90%+ 带宽利用、795μs P99 RTT。基于 Valiant Load Balancing（VLB）"弹跳"思想。微软 94000 服务器实测：过订阅 fat-tree 即便 20% 平均利用，ECMP 下 P99 RTT 仍达 2014μs。
2015 · Presto (SIGCOMM)
把流切成 64KB flowcell 喷射，flowcell 级负载均衡。
2017 · DRILL (SIGCOMM)微秒级 per-packet 随机路由，基于本地队列占用。
处理非对称 Clos（故障/稀疏导致）：将路径分解为最小对称组，组内 ECMP 哈希、组间 per-packet 均衡；非对称越严重越退化为 ECMP。Verilog 实现，硬件开销 <1%。
2019 · FatPaths (ETH SPCL)flowlet 级负载均衡，弹性流大小；证明低直径拓扑（Slim Fly/Xpander）+ FatPaths 在等成本下较 fat-tree 高 15% 吞吐、低 2× 延迟。
2025 · RailS (arXiv)面向 AI all-to-all 的 rail 拓扑负载均衡；理论上证明
完全绕开 spine、仅用 N 条直连 rail 是 all-to-all 最优策略，激活最多 64 个 QP。
2025 · Direct-Connect Topologies (NSDI)算法框架：给定集群规模与度，合成多种直连拓扑与通信调度，按负载选最优。证明
无普适最优拓扑，需 workload-aware 选择。
2026 · Oblivious LB for Sparse Traffic (INFOCOM)证明 VLB 在
稀疏流量下次优，构造最优 oblivious 方案；给出非 oblivious 与 oblivious 路由间 √2 倍最坏负载差距。
3.1 ECMP 的"原罪"
ECMP chooses the next hop of a packet by hashing the five-tuple ... due to hash collision, ECMP cannot achieve full bisectional bandwidth. As reported ... ECMP can only utilize 40-80% network capacity.
Consider a 3-stage fat-tree ... with 27k hosts performing a data shuffle ... hash collisions will reduce the network's bisection bandwidth by an average of 60.8%.
⚠️ 关键判断 稀疏 Clos（过订阅）会放大 ECMP 的缺陷——路径更少、冗余更低，hash 冲突的代价更严重。因此稀疏化必须伴随动态负载均衡，否则性能崩塌。这是 Sparse Clos 设计的"配套刚需"。
4工业界实践：Hyperscaler 的拓扑演进
4.1 Google：从 Clos 到 OCS 直连（最彻底的稀疏化）
2015 · Jupiter Rising (SIGCOMM)3 级 Clos 无阻塞 fabric + 集中式 SDN 控制面 Orion。奠定 hyperscaler Clos 部署范式。
2022 · Jupiter Evolving (SIGCOMM)Clos → 直连 + OCS，
首次移除 spine 层。
5×速度/容量
-30%capex
-41%功耗
3×更快重构
6 Pb/s聚合带宽
Google 强调：直连 fabric 配合流量+拓扑工程，吞吐与 Clos 持平：
We show that the combination of traffic and topology engineering on direct-connect fabrics achieves similar throughput as Clos fabrics for our production traffic patterns.
4.2 AWS：准随机扁平 RNG（结构稀疏 + 随机化）
-69%路由器数量
+33%吞吐
-40%设备电耗
9-45%成本节省
RNG 的弹性优势：故障下按比例降级（1% 路由器损失 ≈ 1% 容量损失），而非 fat-tree 的灾难性区域切断：
Random networks also demonstrate excellent resilience ... The loss of 1% of routers results in a roughly 1% capacity loss. Degradation is proportional and predictable rather than catastrophic and concentrated.
路由创新 Spraypoint：源路由器向所有邻居随机喷射，再绕指定 waypoint 路由，提供近 2× 独立路径：
By spraying to neighbors, Spraypoint provides nearly twice as many independent paths between routers as standard shortest-path routing techniques.
部署时间线：2024 年底都柏林首个生产网络上线，2026 年 4 月成为大多数新建 AWS DC 默认架构。
4.3 Meta：分层过订阅 Clos + DSF 调度 fabric
Meta 的 DCN 从 F4 演进到 F16 Clos，再到面向 AI 的 DSF（Disaggregated Scheduled Fabric, 2025.10）。其 Llama4 训练集群（arXiv 2510.20171, 2026.01）是分层过订阅稀疏 Clos的典型：
The network within a DC adopts a 3-layer Clos architecture ... The Rack Training Switch (RTSW) connects GPUs within a rack, while the Cluster Training Switches (CTSW) connect all racks within an AI Zone. Aggregator Training Switches (ATSW) connect CTSWs across the DC ... Compared to the Llama 3 network, we reduced the cross-AI-Zone over-subscription ratio from 1:7 to 1:2.8 to provide higher bandwidth and better support multi-dimensional parallelism training at larger scale.
💡 稀疏化的"反向调整" Meta 从 Llama3 的 1:7 过订阅收紧到 Llama4 的 1:2.8——并非一味稀疏，而是因多维并行（TP/PP/DP/EP）规模扩大，跨区带宽成为瓶颈，稀疏度需随负载特征动态调优。这印证了"无银弹"判断。
Meta 在过订阅 Clos 上的集合通信优化：farthest-first 递归倍增 all-gather 最优；跨区用 Ring 算法限制并发 in-flight 流量；DQPLB + VOQ 调参使交换机 buffer 积压较 Llama3 降一个数量级。
4.4 Alibaba：3-Tier Clos + C4P 流量工程
These NICs are integrated into a 3-Tier Clos network (Clos, 1953), configured in a Fat-Tree topology with a 1:1 oversubscription rate ... the cluster is capable of supporting over 10,000 GPUs. In a single pod ... it can accommodate 512 GPUs.
Alibaba 的实验直接验证：在 2:1 过订阅（稀疏）Clos 上，C4P 全局流量工程使 allreduce 吞吐提升 65.55%（1:1 时为 70.3%）——证明流量工程能补偿稀疏化带来的带宽损失：
Despite these variations, the overall throughput is substantially improved by 65.55%, and the long-tail problem is obviously alleviated, with C4P enabled.
4.5 NVIDIA：Rail-Optimized / Rail-Only + PXN
NVIDIA 的 Rail 架构是 AI 训练专用稀疏 Clos：第 n 号 NIC 连同一 leaf，形成 N 条并行 "rail"，spine-leaf 仍全连但流量绕开 spine。PXN（PCI×NVLink, NCCL 2.12）用片内 NVLink 减少穿 spine 流量：
Rail-optimized network topology helps maximize all-reduce performance while minimizing network interference between flows. It can also reduce the cost of the network by having lighter connections between rails.
PXN combines NVLink and PCI communications to reduce traffic flow through the second-tier spine switches and optimizes network traffic. It also improves message rates by aggregating up to eight messages into one.
5拓扑对比分析：Sparse Clos 的定位
5.1 主流替代拓扑谱系
| 拓扑 | 核心思想 | vs Fat-tree（Clos） | 来源 |
|---|
| Jellyfish | 随机正则图（RRG） | 等成本下多承载 25% 服务器；平均路径 <2.7 跳 vs ~4 跳；可增量扩容（仅重接 2× 新增端口） | NSDI 2012 |
| Dragonfly | 低直径分层 | >16K 节点时较 folded Clos 省 52% 成本；但二分带宽低、依赖自适应路由 | ISCA 2008 |
| SlimFly | 直径 2，逼近 Moore Bound（MMS 图） | 较 fat-tree 省 50% 成本/能耗；较 Dragonfly 省 25%；路由器减 50%、线缆减 30% | SC16 |
| Direct-Connect | 无交换机直连，按负载合成 | NSDI'25 证明需 workload-aware，无普适最优 | NSDI 2025 |
| Twisted Torus | 扭曲环面 | Google TPU v4 采用，≤10⁴ 节点带宽最优 | NSDI 2025 引用 |
| Zcube / Zettafly | 面向 Zetta 级定制 | 10 万+ 加速器 fat-tree 需数十亿美元；推荐小→Zcube、中→Fat-tree、大→Zettafly | 综述 2026 |
5.2 关键量化对比
at the same cost as a fat-tree with 16,000 servers, Jellyfish can support >20,000 servers at full bisection bandwidth.
DragonFly, which reduces cost by 20% compared to Flattened Butterfly and by 52% compared to folded Clos networks such as fat tree for more than 16K nodes.
Lower diameter ... Cost and energy savings: Up to 50% over Fat Tree, Up to 33% over Dragonfly ... 'It's the diameter, stupid'.
Slim Fly is ≈2× more cost- and power-efficient than fat trees and Clos while offering ≈25% lower latency.
5.3 稀疏 Clos vs 低直径拓扑：本质权衡
💡 研判 Sparse Clos 与 Jellyfly/SlimFly/Dragonfly 处于同一设计空间——都是"用更少互联资源换成本/功耗"。差异在于：
• Sparse Clos 保留 Clos 层次结构，通过减 spine/过订阅实现，工程成熟、可渐进演进（Google/Meta/Alibaba 路线）；
• 低直径拓扑 抛弃层次，靠图论结构降直径，理论更优但需自适应路由，大规模部署受限。
2026 综述指出："高 radix 拓扑（Dragonfly/SlimFly/Galaxyfly）较 Fat-tree 有更高 Moore-bound 效率、更小二分带宽、更低成本，但严重依赖高效自适应路由——这是限制其大规模部署的关键因素。"
They exhibit higher Moore-bound efficiency, smaller bisection bandwidth, and lower cost compared with Fat-tree networks, but rely heavily on efficient adaptive routing algorithms to realize their full performance potential -- a key factor that limits their large-scale deployment.
6AI 训练负载与稀疏 Clos 的深度交互
AI 大模型训练（尤其 MoE）的通信模式——all-to-all——是驱动 Sparse Clos 演进的核心力量。理解这一交互，需站在模型/系统协同设计视角。
6.1 all-to-all：O(N²) 的根本挑战
all2all entails communication from each process to every other process. In other words, the number of messages exchanged as part of an all2all operation in an N-GPU cluster is O(N²).
all-to-all 无法用 tree/ring 优化（二者针对 allreduce 的 O(N) 通信），消息数随 GPU 数平方增长，直接压满任意稀疏互联的二分带宽。
6.2 MoE 稀疏专家流量：ECMP 的灾难
In Spine-leaf networks, ECMP selects cross-leaf paths through static hashing, which performs well for uniform traffic but fails under MoE all-to-all patterns. Sparse expert activations create highly imbalanced traffic that ECMP cannot adapt to, causing NIC and uplink load disparities up to 5–10×.
all-to-all occupies 40%–60% of the iteration time ... RailS improves bus bandwidth by 20%–78% and reduces completion time by 17%–78%. For Mixtral workloads, it shortens iteration time by 18%–40%.
⚠️ MoE 与稀疏 Clos 的张力 MoE 的 all-to-all 既有"通信稀疏"（每 token 只去少数专家）又有"流量不均衡"（专家热度不均）。这与 Clos 的 ECMP 静态哈希根本冲突——稀疏化（路径更少）会进一步放大不均衡。RailS 的结论：完全绕开 spine、只用 N 条直连 rail 是 all-to-all 最优，本质是把"稀疏流量"路由到"稀疏直连拓扑"上。
6.3 直连拓扑对集合通信的适配
NSDI 2025 指出：全阻塞交换网络的经典集合算法（recursive-doubling、Bruck、BlueConnect）不能直接用于度受限直连（稀疏）网络：
Bandwidth and latency optimal collectives for switch networks such as recursive-doubling, Bruck algorithm ... are unsuitable for direct-connect networks, because their one-to-one communication patterns fail to utilize all available links, and they assume a fully connected network.
并揭示度受限拓扑的 Pareto 权衡：低直径利于 all-to-all 吞吐与小数据延迟，但损害大数据 allreduce 的负载均衡：
minimizing the topology's diameter is ideal not only for latency-sensitive allreduce at small data sizes but also for all-to-all throughput; however, this could come at the cost of load imbalance across links in bandwidth-sensitive allreduce at large data sizes.
6.4 成本压力：为什么必须稀疏化
$200M30k GPU Clos（400G）
$153M32k GPU 网络
4.7 MW32k GPU 网络功耗
$300M65k GPU 网络
building a Clos network with high bandwidth to interconnect over 32,000 GPUs would cost $153 million, and the network all by itself would consume 4.7 megawatts ... a full bisection bandwidth Clos fabric linking 30,000 GPUs using 400 Gb/sec links would cost $200 million.
For zettascale computing and interconnecting systems involving over one hundred thousand accelerators, a non-blocking Fat-tree structure would incur interconnection costs of several billion dollars.
7关键挑战与开放问题
| 挑战 | 本质 | 现状/进展 |
|---|
| 最优稀疏度确定 | 过订阅比选多少？随负载变 | Meta 1:7→1:2.8 动态调；INFOCOM'26 oblivious LB 理论下界 |
| 降级态路由 | 稀疏化后冗余低，故障影响大 | DRILL 非对称 Clos 优雅退化；Alibaba C4P 链路隔离+全 mesh 探测；AWS RNG 比例降级 |
| 重配置策略 | OCS/动态拓扑何时重构 | Google OCS 3× 更快重构；TROD 基于 recurring 流量模式；OpenOptics 2μs 电路周期 |
| 线缆复杂度 | 稀疏/随机拓扑布线难 | AWS ShuffleBox 无源光器件；patch panel 比交换机/光模块便宜（NSDI'19） |
| 增量扩展 | Clos/fat-tree 离散规模，扩容需换全部交换机 | Jellyfish 仅重接 2× 新增端口；Google OCS 增量交付；Meta 跨楼扩展 |
| AI all-to-all 适配 | O(N²) 消息压满二分带宽；MoE 不均衡 | Rail-Only/RailS 绕 spine；PXN 减 spine 流量；NSDI'25 workload-aware 拓扑 |
| 故障定位 | 大规模训练故障致 GPU 浪费 | C4 快速定位 faulty 组件；Meta 网络配置错误致 10.7% GPU 任务失败 |
A Meta study found that network configuration errors caused 10.7% of significant job failures in their GPU clusters, with topology-dependent congestion contributing to performance variability.
Network topology design is an engineering art of balance and compromise ... Neither the lowest-cost topology nor the highest-performance, high-cost topology necessarily represents the optimal choice.
8趋势研判与结论（系统/芯片视角）
8.1 三大演进趋势
趋势一：从"静态全阻塞 Clos"走向"动态稀疏可重构 fabric"
Google（OCS 直连）、AWS（RNG 准随机）、Meta（DSF 调度 fabric）殊途同归——拓扑不再是静态铜/光布线，而是按流量需求动态重构的资源。OCS/光交换 + SDN 是使能器。这意味着网络拓扑设计与流量工程将深度融合。
趋势二：AI 负载倒逼"spine 消亡"
LLM 训练流量绝大部分留在 rail 内（leaf 层），spine 利用率极低却成本/功耗巨大。Rail-Only、RailS、PXN 都在系统性地绕开或消除 spine。对芯片设计者：片上交换（如 NVSwitch）正在吞噬 spine 层的功能，GPU/NIC/交换芯片的边界在重新划分。
趋势三：随机化/低直径拓扑从学术走向生产
Jellyfish（2012）的随机图思想 14 年后在 AWS RNG 落地生产。SlimFly/Dragonfly 的低直径优势随 radix 上升愈发明显。但"依赖自适应路由"仍是大规模部署的最后障碍——这恰是芯片层面（交换 ASIC 自适应路由能力）的机遇。
8.2 对芯片/系统研究者的启示
✅ 启示一（交换芯片） 高 radix 交换 ASIC 是稀疏化的对冲——radix 越高，给定端点数下网络直径越小。但单芯片 radix 受光模块/功耗/良率限制，晶圆级/光电混合封装交换可能是下一代突破口。
✅ 启示二（光交换） OCS（MEMS/wavelength-selective）是动态稀疏的物理基础。OpenOptics 实现 2μs 光电路周期，Google Jupiter 证明 OCS 可规模化。光电混合交换芯片（电交换做快速 per-packet，光交换做慢速拓扑重构）是明确方向。
✅ 启示三（GPU/NIC 协同） Rail-Only 依赖片上 NVSwitch 替代 spine；AMD 因无 Infinity Fabric switch 而 Rail-Only 受限。NIC/交换芯片/GPU 互联的垂直整合程度直接决定可稀疏化空间——这是 NVIDIA 的护城河，也是国产算力的卡点。
✅ 启示四（算法-拓扑协同） NSDI'25 证明无普适最优拓扑，需 workload-aware。MoE 的稀疏 all-to-all 与稀疏 Clos 是天然匹配——稀疏路由（MoE expert routing）与稀疏网络（sparse Clos）的联合设计是开放前沿。
8.3 一句话结论
Sparse Clos 不是一种拓扑，而是一个方向——在 AI 万卡/十万卡时代，用"动态可重构 + 稀疏互联 + 片上交换替代 spine + 自适应路由"重新定义数据中心网络，以应对全阻塞 Clos 不可持续的成本/功耗/扩展性。Google 走 OCS 直连、AWS 走准随机扁平、Meta 走分层过订阅，共同宣告了"静态 fat-tree 一统天下"时代的终结。
★深化调研补遗（v2 新增）
本部分整合了首版后的针对性深化调研发现，覆盖 AWS RNG 完整叙事、Meta DSF 解聚架构、Llama4 三层 Clos 定量数据、芯片层（交换 ASIC radix + OCS 规格）与 Zetta 级定制拓扑。
D1. AWS RNG：从 Penrose 瓷砖到"just be random"
arXiv 2604.15261（2026 年 6 月）正式发表，作者包括 Giacomo Bernardi（AWS principal scientist）、Ratul Mahajan（Amazon Scholar/UW）、Seshadhri Comandur（Amazon Scholar/理论CS）等十位 Amazon/AWS 研究人员：
We design and deploy in production the first flat datacenter networks. Our design, called RNG, is based on quasi-random graphs. RNG has a new distributed routing protocol that exploits the properties of random graphs to find a large number of edge disjoint paths between pairs of endpoints. It uses a novel passive optical device that internally shuffles cables, which makes its cabling complexity similar to that of fat trees. We show that RNG matches or exceeds the performance of fat trees for a range of traffic patterns, despite being up to 45% cheaper. RNG is now the default datacenter network for most workloads at Amazon.
James Hamilton 历史叙事（AWS VP 个人博客）：揭示了完整的理论谱系——1970s expander 图（Valiant 1976、Alon-Boppana、Lubotzky-Phillips-Sarnak optimal expanders）→ 1991 Friedman 随机图近似最优 → 2009 VL2（Greenberg 等，fat-tree + VLB，2019 SIGCOMM 时间检验奖）→ 2012 Jellyfish（NSDI）→ 2023-2024 起源故事：Bernardi 从 Penrose 瓷砖（彭罗斯密铺）开始，半年后碰壁，"just be random!" 成为内部笑话。
💡 Hamilton 设计哲学 VL2 把随机化放在协议层（VLB），RNG 把随机化下沉到链路硬件层。"Both approaches are fairly effective at avoiding hot spots but the RNG approach is much more hardware efficient. Perhaps the lesson is to pull optimizations down to the lowest level possible to avoid redundancy not fundamentally required by the approach." 此外，RNG 的增量扩展能力改变了超大规模数据中心的土地/分期/资本部署/变现逻辑——不仅是网络革新。
D2. Meta DSF：解聚 vs 稀疏
Meta 官宣于 2025.10 的 DSF（Disaggregated Scheduled Fabric）并非传统"稀疏 Clos"，而是对交换机内部分解聚的创新：
The fundamental innovation of DSF lies in its two-domain architecture, which separates the network into the Ethernet domain, where servers and traditional networking protocols operate, and the "fabric" domain, where packets will be broken into cells, sprayed across the fabric, and subsequently reassembled at the hardware before being delivered back to the Ethernet domain.
| 组件 | 名称 | 角色 |
|---|
| RDSW | Rack Disaggregated Switch (Interface Node) | 面向外部，路由/L3 |
| FDSW | Fabric Disaggregated Switch (Fabric Node) | 内部 cell spraying，无 L3 |
| SDSW | Super DSF Spine | 跨 DSF 集群互联 |
关键机制：Packet Spraying 替代 ECMP（硬件端 cell 重组保序）→ Credit-based 拥塞控制 → Input Balanced Mode（故障下自动均衡，确保零过订阅）。DSF 对外表现为"单一虚拟交换机"，可突破单体 chassis 的物理端口/带宽极限。
D3. Llama4 三层 Clos：1:2.8 过订阅的确认
Meta NCCLX/Llama4 论文（arXiv 2510.20171, 2026.01）确认为三层 Clos：
The network within a DC adopts a 3-layer Clos architecture. Each DC is partitioned into multiple AI Zones. The Rack Training Switch (RTSW) connects GPUs within a rack, while the Cluster Training Switches (CTSW) connect all racks within an AI Zone. Aggregator Training Switches (ATSW) connect CTSWs across the DC. Compared to the Llama 3 network, we reduced the cross-AI-Zone over-subscription ratio from 1:7 to 1:2.8 to provide higher bandwidth and better support multi-dimensional parallelism training at larger scale.
关键判断：Llama3→Llama4 并不是"更稀疏"——而是收紧过订阅保跨区带宽。跨 DC 使用 ATSW 间全 mesh，同一 1:2.8 过订阅比。拓扑感知调度器分配 rank 到最近节点；过订阅网络下 farthest-first 递归倍增 all-gather 经验证最优；Ring 算法约束跨区并发流量；DQPLB+VOQ 较 Llama3 降一个数量级的 buffer 积压。
D4. 芯片层：radix 翻倍与 OCS 动态化
51.2TTH5 (5nm, 64×800G)
~102.4TTH6 下一代
136×136Google OCS 端口数
1.5msMEMS OCS 切换
Broadcom Tomahawk 5：51.2Tbps 单片 ASIC、5nm 工艺、512 lane 112G PAM4 SerDes、~250ns 转发时延。CPO 变体 Bailly 集成 8×6.4Tbps 硅光引擎，光互连功耗较可插拔降 70%。下一代 TH6 向 102.4T 迈进。
Google OCS（arXiv 2208.10041）：3D MEMS 架构，136×136 端口、毫秒级切换（典型 1.5-3ms、100 端口 600μm 镜面达 1.5ms）。镜面稳定后零额外功耗 + 零排队延迟。SemiAnalysis 估计 Google Apollo/OCS 已节省 >30 亿美元。OCP 已成立 OCS 子项目（2026 白皮书），从 Google 独家走向行业标准化。
💡 趋势 交换芯片 radix 翻倍（51.2T→102.4T→204.8T）+ CPO 硅光引擎 + OCS 光电混合交换 = 物理层面的"spine 消亡"。电交换做 per-packet（~ns 粒度），光交换做拓扑重构（~ms 粒度），二者结合是实现动态稀疏化的物理基础。
D5. Zcube/Zettafly：三规模推荐体系
2026 综述（国防科大/中国）给出定量比较后的推荐：
Based on publicly available pricing information, this study models each topology's per-endpoint copper/optical cable count, port count, cost, and power consumption, ultimately recommending the use of Zcube, Fat-tree, and Zettafly for building small-, medium-, and large-scale interconnection systems, respectively.
Zcube（SIGCOMM 2025 正会，DOI: 10.1145/3718958.3750503）："From ATOP to ZCube: automated topology optimization pipeline and a highly cost-effective network topology for large model training"——面向大模型训练的自动化流水线生成成本效益拓扑。
Zettafly：面向 Zetta 级规模定制。此外 HammingMesh（DL 推理）、HPN7.0（阿里高性能智能计算网络）也属于"目标应用通信模式推导的定制结构"。这三类智能计算拓扑的共性哲学是 workload-aware / 应用定制，与 Sparse Clos 的稀疏化方向一脉相承。