HPC file system supplier Quobyte has added a plug-in for TensorFlow that by-passes the Linux kernel to speed machine learning.
TensorFlow machine learning apps that apply inference and model training on servers with attached GPUs can use the plug-in to go direct to Quobyte files.
Berlin-based Quobyte provides scale-out, parallel access, distributed Data Centre File System software, supporting file, object and block access, including S3 and HDFS. The software supports and optimises the use of NVMe, SAS and SATA SSDs and hard disk drives. The company said its software is self-healing and fault-tolerant.
Customers include Airbus and the UK’s JASMIN supercomputer organisation,
The TensorFlow plug-in means old versions of Linux, with no TensorFlow support, can be used for machine learning. Also the plug-in can be used with the Google Cloud Platform for model training and inference there.
The company said it provides high throughput performance line speed of up to 10GB/sec or 100Gbit/s to each client to keep multiple GPUs per commodity server busy. It can scale out from four to thousands of servers.
When ML workloads are sensitive the Quobyte software features uniform ACLs, X.509 certificates, multi-tenancy, and physical separation of data/workloads on hardware for isolation. It also supports encryption.
Blocks & Files thinks other HPC parallel and scale-out file system suppliers may add the TensorFlow support. Elastifile, Isilon, Lustre, Panasas, Qumulo, Spectrum Scale and WekaIO are likely candidates.
