Networking in the Age of Inference with Hedgehog

Hedgehog explained how the shift from AI training to inference is reshaping the data center network. Marc Austin, Manish Vachharajani, and Sergei Lukianov covered disaggregated inference, which splits prefill and decode across different accelerators, the role of KV cache transfers over a scale-out RDMA Ethernet fabric, and multi-plane back-end designs that keep online inference clusters available. The session took place at Networking Field Day 41, with Austin, Hedgehog’s founder and CEO, setting up the business case before handing off to Vachharajani, the company’s CTO.

Austin framed the session around goodput, meaning time to first token and token throughput, and pointed to a Computex demonstration by customer SambaNova in which a sample workload took 90 seconds on an NVIDIA B200 alone but 30 seconds when paired with an air-cooled SambaNova SN40 RDU over a Hedgehog AI network. Vachharajani explained why: prefill is compute-bound, decode is limited by memory bandwidth, and a single model-in-a-box layout cannot optimize for both. Splitting them across separate pools, however, means shipping the KV cache from prefill to decode. At roughly 320 KB per token for Llama 3, a 32K-token prompt produces about 10 GB, and that transfer sits directly on the user’s response path, taking about 860 milliseconds at 100 Gbps, 215 milliseconds at 400 Gbps, and 100 milliseconds at 800 Gbps per GPU.

The result, Vachharajani argued, is that inference now needs the full scale-out RDMA back-end network once reserved for training, plus a KV cache storage tier, and unlike training it must stay online around the clock. Faster B300 NICs with 800 Gbps breakouts push designs toward multi-plane fabrics, two disconnected Clos networks that libraries such as UCX and NIXL can route across, so an entire plane can be taken offline for maintenance at reduced performance. Hedgehog is designing a successor cluster that adds a converged tier connecting both planes to storage and accelerators that cannot handle multi-plane networking, a topology outside standard reference architectures. Because pairing GPUs with decode accelerators from SambaNova, Groq, or Cerebras requires an interoperable interconnect, Vachharajani said that interconnect today is Ethernet with RoCE, which puts the network at the center of disaggregated inference.