GPU Memory Offload for LLM fine-tuning and inference with Phison aiDAPTIV+

With aiDAPTIV+, Phison makes on-premises AI processing more accessible and affordable, especially for small and medium-sized businesses, government entities, and educational institutions. CTO Sebastien Jean explained that the primary goal of Phison’s solution is to facilitate fine-tuning of large language models (LLMs) on-site. Fine-tuning often demands significantly more memory than inference, making it expensive and difficult for those without massive budgets or a lot of resources.  The presentation highlighted the massive memory requirements for fine-tuning, which can be up to 20 times the memory needed to run an LLM, driving up costs and making it impossible for some organizations to begin with this approach.

Phison’s solution addresses this challenge by decoupling compute and memory. Sebastien Jean, Phison’s CTO, focused on how Phison’s technology, with its AI-optimized SSDs and middleware, enables on-site LLM training and inference. The product uses a combination of their proprietary middleware, Adaptive Link, and custom-built ProSuite software to manage and extend the memory available to PyTorch, effectively turning an SSD into an extended memory pool. This architecture allows for training large models using fewer GPUs.  The system uses a software layer within PyTorch that intercepts calls and then offloads slices of the model to the SSD, which helps in memory management.

By leveraging SSDs and their proprietary controller technology, Phison offers a cost-effective alternative to expensive GPU-intensive setups and targets the SMB, government, and education markets with this solution.  The presentation concluded with a focus on the financial benefits and the sustainability of the solution. By allowing for more efficient hardware utilization, Phison provides not just a financially smart solution but one with power and cooling benefits as well.  Also, by using repurposed NAND, the solution can increase the lifespan of hardware, reduce electronic waste, and extend the useful life of data center infrastructure.