MinIO Presents at Cloud Field Day 23

Introducing MinIO AIStor – Object Storage for AI and Analytics with MinIO

Watch on YouTube
Watch on Vimeo

MinIO’s VP of Product Marketing, Jason Nadeau, introduces the AIStor object storage solution, designed for AI and lakehouse analytics environments, at Cloud Field Day 23. AIStor distinguishes itself from object gateway approaches by being object-native. Nadeau highlights the importance of object storage for AI, as evidenced by its use in building large language models and various data lakehouse tools. In contrast to the complex, multi-layered architecture of retrofit object gateway solutions, AIStor presents a simpler, direct-attached architecture, leading to superior performance, data consistency, and scalability.

Nadeau emphasizes MinIO’s object-native architecture, which provides strict consistency and SIMD acceleration, resulting in significant performance advantages. These architectural benefits translate into tangible storage outcomes, allowing customers to scale from petabytes to exabytes.  AIStor’s architecture facilitates real-time data services even during hardware failures. This object-native approach enables optimal hardware utilization and cost-effectiveness. MinIO offers direct engineer support, bypassing traditional support queues and providing customers with direct access to experts. The company is seeing strong enterprise adoption and growth in headcount.

The presentation features examples of AIStor deployments in various use cases, including generative AI, high-performance computing, data lakehouses, and object-native applications, as well as an autonomous vehicle manufacturer, a cybersecurity company, and a fintech payments provider. These deployments are achieving desired performance and are helping to control costs.  MinIO plans to offer a channel bundle skew, which will simplify the acquisition of AIStor by bundling hardware and software into a single SKU.

Personnel: Jason Nadeau

AIStor – PromptObject, AIHub, and MCP Demos with MinIO

Watch on YouTube
Watch on Vimeo

Dil Radhakrishnan presented MinIO’s AIStor capabilities at Cloud Field Day 23, focusing on how MinIO is adapting to AI workloads. The presentation demonstrated three key features: AI Hub, PromptObject, and Model Context Protocol (MCP) server. AI Hub provides a Hugging Face-compatible repository for securely storing private AI models and datasets within the AIStor environment. This enables developers to manage and deploy fine-tuned models without exposing them to the public, leveraging the familiar Hugging Face ecosystem.

The presentation then introduced PromptObject, which enables interaction with objects in AIStor using large language models (LLMs). By integrating GenAI capabilities directly into the S3 API, developers can use the “prompt” function to have the LLM extract specific data from unstructured objects, transforming it into structured JSON for easier application integration. This approach eliminates the need for separate RAG pipelines in many scenarios, as prompt objects simplify the process of interacting with single objects. Still, it can also be used in combination with a RAG implementation.

Finally, the presentation showcased the AI Store MCP server, which enables agentic workflows. The MCP server allows AI agents to interact with the data stored in MinIO. This was demonstrated using a cloud desktop, showing how an agent can list buckets, extract information from images, automatically tag data, and create visualizations of the AI Store cluster. This approach enhances data accessibility and facilitates automation in managing and analyzing data.

Personnel: Dil Radhakrishnan

MinIO AIStor, S3 Express API, NVIDIA GDS,m and BF3 Overview with MinIO

Watch on YouTube
Watch on Vimeo

AB Periasamy, Co-CEO of MinIO, presented at Cloud Field Day 23, focusing on the AI-centric capabilities of MinIO AIStor. The presentation highlighted three key areas: S3 Express API compatibility, integration with NVIDIA GPUDirect Storage (GDS), and the forthcoming integration with NVIDIA BlueField 3 DPUs. These technologies aim to enhance performance and efficiency for AI and data-intensive workloads.

The discussion began with S3 Express, a refined subset of the Amazon S3 API designed for high-performance applications, particularly those involving AI workloads. MinIO has implemented the S3 Express API, offering users the choice between the regular S3 API and the S3 Express API, without requiring changes to data formats. The presentation emphasized that S3 Express eliminates performance bottlenecks, such as directory sorting and unnecessary checksum computations, that limit modern AI applications. It provides faster time-to-first-byte metrics compared to traditional S3.

Next, Periasamy introduced GPU Direct, an NVIDIA interface that allows direct data transfer between storage and GPU memory, thereby bypassing the CPU. The upcoming integration with MinIO will enable this functionality using the S3 API. This is done by utilizing a control plane via HTTP with RDMA as a data channel. The presentation concluded with a discussion of how integrating with NVIDIA BlueField 3 DPUs would enable an ultra-efficient JBOD-based storage solution, also known as JBOF. This new design will result in a low-power solution with high performance due to being a solely smart NIC-based system.

Personnel: AB Periasamy