Hammerspace and Parallel Works are collaborating on a combined offering enabling IT infrastructure, research computing, and AI teams to orchestrate compute, storage, and data workloads across sites and clouds, and scale cloud compute resources for burst capacity.
Hammerspace is a data orchestration company. Its Global Data Platform (GDP) uses parallel NFS-based file system software to manage data in globally distributed locations, both in file (NFS) and object storage, using SSDs, disk drives, public cloud services (AWS, Google Cloud, Azure, and Seagate Lyve Cloud), and tape libraries. The data is located in a global namespace and can be orchestrated, placed, and accessed as if it were local.
Parallel Works, grown out of Argonne National Laboratory, has developed ACTIVATE control plane software to provision, manage, and share HPC and AI IT resources across on-prem locations, public clouds, and hybrid environments. It supports cross-cloud clusters and separate compute and storage provisioning.
David Flynn, Hammerspace founder and CEO, stated: “Seamlessly orchestrating both compute and data across locations and cloud regions is a game-changer. Our unified solution with Parallel Works simplifies operations, reduces overhead and accelerates innovation by ensuring data is always where it needs to be, with minimal latency.
“Innovators need the agility to move complete workloads across different compute clusters – adding flexibility to use CPUs for data preparation, GPUs for processing, or accessing remote clusters for extra cycles or cost-efficient cloud regions.”
Parallel Works CEO Matthew Shaxted said: “By integrating our expertise in HPC and AI workload orchestration with Hammerspace’s parallel file system, global namespace and data objectives, we’re enabling application portability and achieving agile task placement. This approach empowers organizations to efficiently allocate resources where needed, driving performance, innovation, and simplified hybrid infrastructure management.”
The two companies say that AI and HPC workloads can demand significant compute resources and low-latency access to large datasets. Such data and compute resources are often distributed across different locations. Do you move the compute to the data or vice versa?
Also, if you need short-term burst compute, using alternate datacenters or cloud infrastructure can provide a scalable, elastic option yet “orchestrating workloads across multiple, geographically dispersed clusters – especially when data is remote from the compute – has been highly complex.”
They pair say: “This complexity arises from the need to relocate the application while managing data transfer, network latency and synchronization, leading to error-prone and time-consuming setups that most teams have been unwilling to take on the burden of managing.”
A combination of Hammerspace’s GDP and Parallel Works’ ACTIVATE can automate the provisioning and management of local and cloud compute resources and orchestrate data flows to them.
ACTIVATE is the unified control plane to create, provision, and manage the distributed compute resources. It automates workflows such as spinning up Slurm instances to manage job scheduling and the associated HPC nodes.
Hammerspace’s GDP functions as a data orchestrating control plane “that provides high-performance file and object storage, creates a global namespace that spans sites and clouds, and automates the flow of data within that namespace with Data Orchestration services. Hammerspace delivers the data to the HPC servers regardless of where they reside, with orchestration policies set at the share, directory, and even individual file levels.”
The two companies say customers can “right size” their infrastructure with the ease of use of a fully managed cloud service but at the cost point of standard cloud infrastructure services, delivering “significant cloud cost savings” versus, for example, managed Lustre.
A cloud cost comparison executive brief can be found here and a joint solution brief doc here. Flynn and Shaxted will discuss their combined offer in a webinar hosted by The Register on October 24. Sign up for the webinar here.
