HPC workloads vary from using very large files, through medium-sized ones to myriad small files and mixed workloads. An HPC storage system must be tuned to match the IO patterns of its main workloads – unless, of course it is bought for one specific type of file IO,
Panasas, the HPC file storage supplier, has devised a workaround with a PanFS update that auto tunes to speed file access with different IO types.In its announcement today, Panasas claims all other parallel file systems require clumsy tiering and-or manual tuning to compensate for specific workload characteristics and changes. That leads to inconsistent performance and higher admin costs.
Panasas’s new feature, called Dynamic Data Acceleration (DDA), does this automatically using the same tiering and caching ideas and does it consistently faster with no need for manual tuning.
With DDA, NVMe SSDs store metadata, lower-latency SATA SSDs store small files, and large files are stored on low-cost, high-bandwidth disk drives. Storage-class memory NVDIMM holed transaction logs and internal metadata, and the most recently read or written data and metadata are cached on DRAM.
Small file IO has more metadata accesses – such as file and folder lookup – as a proportion of an overall file IO than large file IO, where the data portion off the IO can be many times larger than the metadata portion.
PanFS dynamically manages the movement of files between SSD and HDD, and uses fast NVMe SSDs, NVDIMM and DRAM to accelerate file metadata processing. That speeds small file IO much more than it would large file IO, where the metadata processing occupies a much smaller percentage of the overall file IO time.
DDA is included in the latest version of PanFS, which is now generally available to supported customers.
