Watch on YouTube
Watch on Vimeo
VMware by Broadcom’s Elliott Davis, AI architect for VMware Cloud Foundation (VCF), used this Tech Field Day session at KubeCon North America 2026 to explain how enterprises can host AI workloads on VMware vSphere Kubernetes Service (VKS) using open-source components. His central point is that AI agents, rather than humans, will generate most future LLM calls. A single human question can now trigger hundreds of tool calls, retrievals, and model calls as agents plan, retry, and spawn subagents, producing load that Davis compared to a DDoS. He split the problem into two sides: token production on GPUs, which is expensive but measurable through metrics such as time to first token and throughput, and agents, which consume those tokens in bursts and sit idle most of the time while still holding the pod resources they requested. Davis said enterprise agent adoption is still early, and most VCF customers today prefer the platform’s no-code agent builder. He cited a Gartner projection that 90% of enterprise agent workflow steps will run on-premises by 2030 at roughly 20 times lower cost, and stressed that the larger benefit of open-weight models is keeping control of enterprise data.
Davis described a three-layer stack, with VCF contributing at each level. At the bottom, the model runtime pairs an inference engine such as vLLM with Run:ai, which Davis said is now fully open source and handles GPU scheduling, partitioning, and preemption. vSphere adds hardware-level GPU slicing and multi-tenancy, assigning GPU capacity to separate VKS clusters without building separate physical clusters for each team, because on-premises GPUs only pay off when they stay busy. In response to a delegate’s question, Davis placed LLM-D behind the gateway layer for KV cache-aware routing and disaggregated inference, which avoids recomputing the static prompt prefixes common in agent requests. In the middle sits Agent Gateway, the open-source CNCF project from Solo.io, which acts as the policy enforcement point, MCP proxy, and agent-to-agent (A2A) gateway. Istio and Envoy policies force all agent traffic through it, credentials are injected along the way so untrusted agents never hold them, and it decides whether each request goes to an on-premises open-weight model or a frontier model. Broadcom contributes to the project and plans to ship Agent Gateway as a VKS add-on within two weeks. Davis acknowledged that gateway sprawl is an industry problem, with several gateways inside Broadcom alone, and described the pieces as interchangeable building blocks.
The top layer runs the agents themselves through the open-source Agent Substrate project, which treats agents more like network requests than long-running infrastructure. Running one agent per pod would overwhelm a cluster, so Agent Substrate keeps a warm pool of sandboxes and switches agents on and off within roughly 500 milliseconds, multiplexing many agents onto shared resources. Davis has run hundreds of gVisor instances inside a vSphere Pod, with layered isolation: gVisor contains the agent, the VKS node VM or vSphere Pod restricts any escape, and Agent Gateway with deny-by-default network policy blocks outbound access. He closed with a use case drawn from customer requests: an agent that finds orphaned persistent volume claims and idle GPU reservations across VKS clusters and proposes a cleanup plan without acting until a human approves. Listing resources can go to an on-premises model, assessing risk goes through Agent Gateway to a frontier model, the dry run executes in a sandbox, and the agent suspends itself while waiting for approval so it consumes no resources until the user returns.
Personnel: Elliott Davis
Thank you for being part of the Tech Field Day community! Our mailing list is a great way to stay up to date on our events and technical content, and we appreciate your signup.
We promise that we’ll never spam you, send ads, or sell your information. This list will only be used to communicate with our community about our events and content. And we’ll limit it to no more than one message per week.
Although we only need your email address, it would be nice if you provided a little more information to help us get to know you better!