SPEC replaces SPECsfs 2014 benchmark with AI and genomics workouts

One of the more complicated file storage performance benchmarks is being replaced – by a successor that is even more complicated.

The SPECsfs 2014 SP2 benchmark determines file performance in five different workloads: software builds, video streaming (VDA), electronic design automation (EDA), virtual desktop infrastructure (VDI) and database. Each category test gets a numerical score and overall response time (ORT).

The new SPEC Solutions 2020 benchmark modifies the SFS 2104 SP2 test by adding two new workloads, dropping two existing ones, and tweaking the test methodology. This means that the new benchmark results cannot be compared to the old ones.

There is a new AI_IMAGE (AI image processing) workload, representative of AI Tensorflow image processing environments, and a new Genomics one. The existing SPECsfs 2014 software build, EDA and VDA workloads are retained but the VDI and Database workloads are being scrapped.

That’s because the technology of these has changed and is continuing to evolve. These workloads could be re-introduced later, after the collection of new base trace data has been completed.

SPEC Solutions 2020 benchmark scaling diagram.

The test scale has substantially increased. SPECsfs2014_SP2 scaling was up to 60,000 load-generating processes globally. This is expected to increase to around 4 million load-generating processes distributed globally in SPEC Solution 2020. All of the geographically distributed load generating processes will be kept in sync, at a sub millisecond resolution.

The updated benchmark has a new statistical collection mechanism that enables users to extract runtime counter information and load it into a database for graphical representation by, for example, Graphite, Carbon, and Grafana.

SPEC Solution 2020 graphical output.

SPEC Solution 2020 can support a non-POSIX storage system with a plugin shared library for accessing an AFS storage service. Others may be added in the future, such as an object store, S3, Ceph, and more. The new test supports custom workloads for private, no-publishable testing. It’s also possible to combines Unix and Windows load generators in a workload definition. 

A SPEC Solution 2020 What’s New document has lots more information.