Intel Gaudi 3 AI Performance Testing with Signal65

Over the last few years, generative AI has demonstrated its immense potential as a revolutionary technology. AI-powered applications have demonstrated the ability to enhance automation, streamline workflows, and accelerate innovation. Furthermore, the technology has proven to be broadly applicable, with opportunities for creating new, intelligent applications across virtually every industry. While the value of generative AI is apparent, the powerful hardware required to run such applications often serves as a barrier. As AI is increasingly moving from an experimental trend to the backbone of real-world applications, IT organizations are challenged with balancing the necessary performance with economic considerations of AI hardware, and doing so at scale.

Signal65, a performance testing and benchmarking team within the Futurum group, presented their findings on Intel Gaudi 3 AI accelerators at Cloud Field Day 23. The presentation focused on AI inference performance, detailing two main projects: on-premises testing and cloud-based testing on IBM Cloud. The on-premises testing compared Gaudi 3 with NVIDIA H100, using the Kamawaza AI testing suite on Meta’s Llama models (8B and 70B parameters) with varying input/output token shapes. The results showcased Gaudi 3’s competitive performance, especially when factoring in the lower cost, resulting in up to 2.5 times better price-performance than the H100.

The presentation then shifted to Gaudi 3’s performance on IBM Cloud, testing against both H100 and H200. The testing included Granite, Mixtral, and Llama models. Gaudi 3 consistently showed better performance compared to H100 and was very competitive against H200, also showing significant cost advantages, with a 30% lower hourly rate than the NVIDIA options. In both on-premise and cloud scenarios, the speaker highlighted the importance of considering both performance and price when evaluating AI hardware options, particularly for enterprises deploying AI applications at scale. The presentation concluded with a call to recognize the growing competitiveness of the AI hardware market, moving away from a singular NVIDIA dominance.