SPONSORED FEATURE: There’s a pressing need for efficient new data storage solutions given the growing trend of enterprises now deploying AI-enabled applications.
Where megabyte and terabyte storage loads were once commonplace for mere document and single image-type workloads, petabyte (1K terabytes) and even some exabyte (1K petabytes) jobs are now in production.
Factors that have fueled a boom in AI applications include large language models (LLMs) being used in everything from facial recognition software to recommendation engines on streaming services, all to improve user experiences and business processes. Across industries, there’s a growing need for automation, data analysis and intelligent decision-making. AI can automate repetitive tasks, analyze vast datasets to uncover patterns and make data-driven predictions or recommendations. This translates to potentially increased efficiency, productivity and innovation in various fields.
All of this entails vast amounts of data coming from social networks, GPS transmitters, security cameras, point-of-sale locations, remote weather sites and numerous other sources. This trend demands high-performance storage solutions to handle the large volumes of unstructured data involved in AI training and inferencing which can be spread across both on-premises and cloud environments.
A recent IEEE Spectrum report, “Why AI Needs More Memory Than Ever,” explored the ever-increasing data storage demands of AI systems, particularly focusing on the growing size of LLMs. It suggests that besides the demand for high performance, low power, low cost and high capacity, there is also an increasing demand for more smart management functions in or near memory to minimize data movement. As a result, the trend toward deploying hybrid clouds, where all of this is possible, is getting traction.
Traditionally, AI implementation has been marked by siloed solutions and fragmented infrastructure.
“When your applications and tools are running mostly in the cloud, it’s imperative for users to put their data closer to where these tools and applications run,” says Kshitij Tambe, Principal Product Manager at Dell Technologies. “So now if you have your data sitting on premises, and you are building some of these tools and applications to run in the cloud, then there is a big disparity. If you have one thing running in the cloud and enterprise data in the datacenter, this becomes very problematic. So that’s where the need for these hybrid cloud models will come in.”
Why RAGS add even more data to AI systems
They are powerful and require lots of storage, but the LLMs which provide the foundation of AI applications and workloads can only generate responses based on how they’ve been trained. To address this and ensure access to up-to-date information, some AI systems utilize a process called Retrieval Augmented Generation (RAG). RAG integrates information retrieval with prompts, allowing the LLM to access and leverage external knowledge stores. This approach necessitates storing both the base LLM and the vast amount of data it retrieves for real-time use.
With companies – especially long-established ones – building and using many different types of storage and storage devices over years in datacenter, edge and cloud deployments, it becomes a complex problem to managed data across multiple locations at the same time. What some storage admins wouldn’t give to have a single-screen, real-time look at all a company’s storage workloads – whether in production or not – wherever they are in the world!
That was a pipe dream for the longest time. But perhaps not anymore.
New data management platform and processes have emerged in the last year or so to handle these spread-out, next-generation workloads. One example is Dell APEX File Storage in the Microsoft Azure cloud, a NAS platform built to meet AI capacity, performance and data management requirements spanning multicloud environments which is part of Dell’s AI-Ready Data Platform.
Dell APEX File Storage for Microsoft Azure, which became generally available April 9th, bridges the large gap between cloud storage and AI-driven insights, says Dell. It also allows customers a degree of flexibility in how they pay for the service.
At the heart of Dell APEX File Storage for Azure lies PowerScale OneFS, a high-performance scale-out file storage solution already deployed by more than 16,000 customers worldwide.
By bringing PowerScale OneFS to the Azure cloud, Tambe says: “Dell enables users to consolidate and manage data more effectively, reduce storage costs and enhance data protection and security – all while leveraging native cloud AI tools to arrive at insights faster.”
APEX File Storage for Azure serves as a versatile connector to smooth the transition during cloud transformation and enable secure connections to all storage nodes, no matter what type of storage is utilized. A key bonus: the Microsoft interface and control panels have natural familiarity for IT administrators while the PowerScale OneFS replicates the user experience that storage IT professionals are familiar with on-premises.
The APEX File Storage for Azure solution is based on PowerScale OneFS and validated to work with other Dell solutions such as PowerEdge. APEX configurations and specifications include support for up to 18 nodes and 5.6PiB in a single namespace; no other provider can make this claim, boasts Dell. Thus, Dell APEX File Storage for Microsoft Azure puts its stake in the ground with the assertion that it is the most efficient scale-out NAS solution now in the market.
Analysis conducted by Dell indicates that in comparison to Azure NetApp Files, for example, Dell APEX File Storage for Microsoft Azure enables 6x greater cluster performance, up to 11x larger namespace, up to 23x more snapshots per volume, 2x higher cluster resiliency, and easier and more robust cluster expansion.
“Typically, customers might have three nodes, four or five nodes, but there is flexibility to go all the way up to 18 nodes in a single cluster,” says Tambe. “The new architecture of APEX is such that the larger the cluster size, and the larger your data set, it becomes more and more efficient – efficient in the sense of even by the metric of how much usable space you have in your data set.”
Integration and deployment on Microsoft Azure
As for data management, APEX File Storage for Azure offers a new path with integration of high-performance storage capabilities to deploy on Microsoft’s Azure infrastructure. The idea is to let admins easily move data from on-premises to the cloud using advanced native replication without having to refactor any storage architecture. That can deliver huge time savings which subsequently enable data management capabilities to help organizations design, train and run AI-enabled workloads faster and more efficiently, says Dell.
APEX File Storage for Azure leverages Azure’s cloud infrastructure and functionalities to benefit AI tasks in a few ways. Developing infrastructure for advanced AI models necessitates significant investment, extending beyond powerful compute resources to encompass critical data storage infrastructure. Training datasets can range in size from terabytes to petabytes, and concurrent access via numerous processes. Saving checkpoints which each potentially consist of hundreds of gigabytes is equally vital.
APEX File Storage directly integrates with several of the most common AI tools – including Azure AI Studio, to change the way developers approach generative AI applications and help simplify the journey from concept to production. It’s a developer’s playground for evaluating responses from large language models and orchestrating prompt flows says Dell, ensuring optimal performance and scalability.
And since OneFS supports S3 as an access protocol, getting APEX File storage to work with Azure AI Studio should be easy. Developers can point Azure AI Studio using OneLake Data Gateway directly to a OneFS directory, for example. This allows them to use files on OneFS clusters (AFS or on-prem) without copying the data to Blob Storage thus running fine-tuning of AI models with files remaining in a OneFS filesystem.
For providing scalability, APEX File Storage utilizes Azure’s cloud-native technologies, allowing it to elastically scale storage capacity and performance based on AI workload demands. This helps ensure smooth operation, even when dealing with large datasets used in AI training and processing.
For integration, APEX File Storage integrates directly with the Azure architecture, facilitating data transfer between on-premises and cloud environments. This eliminates the need to redesign storage infrastructure when moving AI workloads to the cloud. This combination creates the foundation for a universal storage layer that simplifies storage management in multicloud environments, says Dell.
For data management and protection, APEX File Storage offers features such as advanced native replication, data deduplication and erasure coding. These functionalities assist with data redundancy, security and efficient storage utilization, which are all crucial aspects of managing large datasets for AI applications.
Dell preceded the Microsoft Azure APEX initiative with an AWS version of the service last year. This stands as an example of Dell’s commitment to offering a wide range of storage and data management options for different cloud platforms to meet customer requirements.
Sponsored by Dell.
