Signal65 PINNACLE the Measure of Working Intelligence

Ryan Shrout, who leads Signal65, presented Pinnacle as an innovative approach to performance analysis and testing for AI models. Drawing from his extensive background in measuring CPU and GPU performance, including a tenure at Intel’s competitive analysis group, he highlighted that Pinnacle moves beyond traditional benchmarks that offer mere numbers or speed. The core objective of Pinnacle is to evaluate what AI can deliver, considering its cost, intelligence level, and performance across diverse infrastructure, ultimately aiming to help enterprises select the most suitable models and hardware for correctly completing agentic tasks.

Signal65 Pinnacle differentiates itself by prioritizing “finished work first and then price it,” shifting the focus from simply tokens per second to the actual completion of complex jobs. These jobs often involve numerous model calls and a substantial exchange of contextual tokens. The benchmark employs an execution harness developed with Kamiwaza, which creates a secure sandbox environment for agents to operate, ensuring they are unaware they are under test. Results are deterministically graded against an external, uncompromised answer key, bypassing human or LLM judgment and preventing models from learning or cheating. This methodology allows for the evaluation of model intelligence, silicon capacity, and solution economics, incorporating assessments of agentic task completion and the model’s tendency towards grounding or fabrication. Pinnacle also tests models in both well-structured “governed” and messy “as found” data environments to mirror real-world enterprise conditions.

Through its testing, Pinnacle has revealed significant disparities, such as a hundred-fold difference in the cost of a correct agentic task across various models and a seven-fold variation in service floor based on GPU generation. The platform provides actionable insights, enabling users to choose models based on their specific accuracy and cost-efficiency requirements for different personas, which are built upon generalized work activities outlined by the US Department of Labor. While public metrics are available, Signal65 also offers customized profiles for enterprises seeking more tailored solutions. A notable observation has been the rapid enhancement in model capabilities, with a significant increase in performance and a decrease in error rates over a short period, illustrating that models are becoming both smarter and more economical at an exceptionally fast pace.