Network Computing: How To Cluster 128 Units H100 ?

SINGAPORE - March 4, 2024 - PRLog -- The NVIDIA DGX H100 is equipped with eight single-port ConnectX-7 network cards supporting NDR 400Gb/s bandwidth and two dual-port Bluefield-3 DPUs (200Gb/s) supporting IB/ Ethernet networks.

The DGX H100 is equipped with four QSFP56 ports for storage networks and In-Band management networks. In addition, there is one 10G Ethernet port for Remote Host OS Management and one 1G Ethernet port for Remote System Management.
From the internal network topology of the server in the following figure, there are four OSFP ports for computing network connections (the four in purple), and the blue square is the network card, which acts as both a network card and a PCIe Switch extension function, acting as a bridge between the CPU and the GPU.

Optical module usage: A 400 Gbit/s optical module is required for the downstream port of a Leaf switch. The required value is 32 x 8 x 4. The uplink ports on Leaf switches use 800 Gbit/s optical modules. The required value is 16 x 8 x 4. The downstream ports of the Spine switch use 800 Gbit/s optical modules. Therefore, in a cluster of 128 H800 servers, the computing network uses 1536 800G optical modules and 1024 400G optical modules.

https://www.fiberstamp.com/industry-insights-11163.html