Introducing Arrcus Inference Network – AINF Demo

Nalin Pai of Arrcus introduced the Arrcus Inference Network Fabric, a layer 7 to layer 3 routing fabric that steers inference requests across distributed sites based on model availability, load, and policy. The session covered the AINF router and orchestrator, runtime integrations, network slice support for data sovereignty, and a demo of the management portal, followed by delegate discussion of latency, cost, and target customers. Pai, Arrcus’s chief architect for AI solutions, presented at Networking Field Day 41 with additional input from Keyur Patel and Sanjay Kumar.

Pai positioned AINF as a front-end distribution layer for operators running their own models, sitting behind existing API gateways and alongside runtimes such as vLLM, Dynamo, and SGLang rather than replacing them. The AINF router is a Kubernetes-deployed software service that uses plugins for llm-d and Dynamo to gather KV cache usage, queue depth, worker counts, and GPU class, then shares that data with peer routers so each can pick the best site while the in-site load balancers pick the GPU. A central orchestrator handles lifecycle, policy, and telemetry but stays out of the request path, and a tiered control plane distributes load updates well under a minute. Policies combine match-action routing rules on layer 7 metadata, including PII flagged by lightweight LLM classifiers or third-party tools, with a scored load-balancing step. Operators can also feed WAN slices into AINF, and the demo showed a US-only slice built on SRv6 Flex-Algo, with traffic steered through DSCP or BGP coloring.

Asked whether inspection adds latency, Pai argued that time to first token is measured in seconds, so spending a few milliseconds choosing a site with headroom lowers overall job latency. Arrcus confirmed AINF can run purely as an overlay, keeping load balancing and compute-level sovereignty without network slicing. Kumar tied AINF’s value to faster time to first token, higher tokens per second, and power-aware site selection, and when a delegate questioned whether buyers can span network, cloud, and serving teams, he cited proofs of concept with Lightstorm and TELUS, two others underway, and an unnamed enterprise with distributed factories that needs in-country inference and lacks the power to feed every site.