Weka’s fast filesystem — WekaIO — has added support for Nvidia’s latest offering, the HGX AI-focussed GPU server, shipping file data to its AI-processing GPUs.
This so-called AI supercomputer features an A100 80GB GPU, NDR4 400G InfiniBand, and GPU-Direct Storage software. Weka’s filesystem software can run in an attached Xeon server and ship data over to the HGX using GPU-Direct and avoiding any involvement in data handling by its host Xeon CPU or data staging in the host server’s DRAM.
Liran Zvibel, co-founder and CEO at Weka, issued a quote mentioning the close relationship between Weka and Nvidia: “Nvidia is not only a technology partner to Weka but also an investor. As such, we work closely to ensure that our Limitless Data Platform supports and incorporates the latest improvements from Nvidia as they become available.”
Weka says it recently delivered among the greatest aggregate NVIDIA Magnum IO GPU-Direct Storage (GDS) throughput numbers of all storage systems tested at Microsoft Research. Tests were run on a system that had WekaFS deployed in conjunction with multiple Nvidia DGX-2 servers in a staging environment. Weka says the engineers achieved the highest throughput of any storage solution that had been tested to date.
DDN and VAST Data are also offering HGX support and shipping GDS-supporting software. Excelero, IBM, Micron, Pavilion and ScaleFlux are due to announce availability of their offerings in the next few weeks or months, with Dell EMC, Hitachi, HPE, Kioxia, Liqid, NetApp, Samsung, System Fabric Works, and Western Digital all possibly (and probably) following them.
Nvidia is leading a great herd of storage suppliers to the enterprise AI promised land, where stored data feeds analytic routines for ever and ever.
