Validating Large Al Cluster Fabrics Using Keysight Al Data Center Builder

Keysight aims to help organizations build sovereign AI and neoclouds by addressing critical challenges in validating large AI clusters and their underlying fabrics. The presentation highlights significant inefficiencies observed in AI training tasks, where over 43% fail, with half of these failures (21%) directly attributable to network issues. Such failures lead to expensive checkpoint reloads and idle GPU time, impacting return on investment. Furthermore, even in successful tasks, GPUs spend more than half their time waiting on communication, indicating low utilization and prolonged training times, with only 20% of the time actually dedicated to computation.

To tackle these problems, Keysight introduces its flagship product, Keysight AI Data Center Builder, designed to benchmark, optimize, and co-tune AI fabrics before they are deployed in production. This tool offers three primary applications: collective benchmarks, which emulate single AI collective operations (data movements between GPUs) to provide metrics like network utilization, job completion time, and congestion; a mix of collectives, which takes it further by simultaneously emulating several collective operations across different nodes to assess fabric behavior under complex loads; and workload emulation, utilizing “chakra traces” for sequential model training steps, facilitating regression testing and training time optimization.

The Keysight AI Data Center Builder is available in both hardware (e.g., Aries One chassis) and software versions, and it provides comprehensive insights into fabric performance. It offers key metrics such as collective completion time, algorithm bandwidth, bus bandwidth, and ideal performance percentages, along with duty cycle to indicate GPU utilization. Visualizations like cumulative distribution function (CDF) graphs help identify tail latency, while monitoring Priority-Based Flow Control (PFC) frames indicates network congestion. The tool also provides timelines and heat maps to pinpoint performance outliers and issues like reordered packets. By enabling users to test and optimize their fabric without engaging expensive GPUs, Keysight ensures that the underlying infrastructure is robust and highly efficient, thereby increasing GPU utilization, reducing idle time, and ultimately making AI training jobs complete faster and more reliably.