Connecting the AI Loop with CoreWeave Forge Live Demo

Russ Ratshin from CoreWeave’s product marketing team presented a live demo demonstrating how to connect the AI loop using CoreWeave Forge. He explained that the AI loop is a recognized workflow for developing AI agents, and CoreWeave Forge serves as the environment to facilitate this process. The presentation took the audience through each stage of the AI loop with a practical help desk use case, showcasing the product in action and highlighting its real world applicability for agent building.

The presentation detailed the five stages of the AI loop: run, observe, curate, improve, and evaluate. The ‘run’ stage involves deploying the model for inference and exposing the agent to users. Next, ‘observe’ captures all interaction traces and stores them in Agent Lens, allowing product managers, domain experts, and AI engineers to review the agent’s performance. In the ‘curate’ stage, tagging is used to organize and filter this invaluable production data, making it actionable for further model enhancements and evaluations. The ‘improve’ stage covers various post-training methods, and ‘evaluate’ confirms that the agent model has indeed been significantly enhanced, ultimately leading back to the ‘run’ stage with an improved model.

The demo illustrated these steps using a help desk agent named Harness, which utilizes an LLM served by CoreWeave Inference. It began by instrumenting the agent with ARIA to gather traces for observability within Agent Lens. The Agent Lens dashboard provided a comprehensive view of the agent’s activity, including conversation counts, errors, tags, cost, latency, and model and tool usage. The speaker showed how tags could be created manually or automatically by an LLM judge, with a built-in alignment feature to ensure the judge accurately applies tags. Automated insights were presented, clustering conversations by agent failures and user intents. For the ‘improve’ phase, serverless reinforcement learning was applied, running numerous rollouts against CoreWeave sandboxes to train the agent to make better decisions. Finally, the evaluation phase compared the improved model to the base model, demonstrating superior performance across key metrics before providing the URI of the best model version for redeployment, thus completing the AI loop in CoreWeave Forge, which offers a unified and open environment.