Measure for Measure: SPC-1 benchmark yields new class of storage price-performer

Korea’s Telecommunications Technology Association (TTA) has filed an SPC-1 benchmark ranking seventh in the performance list, and cementing a new performance and price-performance grouping within the SPC-1 rankings.

The organisation filed the benchmark on behalf of Korea’s Taejin Infotech Co. The result, together with a recent filing by China’s Inspur, establishes an informal category for the SPC-1 benchmark: sub-$350/KIOPS at 1.5 million IOPS sub-level.

The Taejin Infotech configuration features two Jet-Speed HHS3124F and five Jet-speed HHS211F nodes, all using NVMe SSD data drives with either SATA SDDs or HDDs as system drives. The nodes use a Mellanox ConnectX-5 EDR 100Gbit/s InfiniBand interconnect.

Made to Measure

The SPC-1 benchmark tests shared external storage array performance in a test designed to simulate real-world business data access. Its main measurement unit is SPC-1 IOPS. These are not literally drive IOs per second but transactions consisting of multiple drive IOs. 

A second measure is price/performance, expressed as dollars per thousand IOPS ($/KIOPS), and then there are response times, application storage unit (ASU) capacity, ASU and the total system price.

TTA used a 7-node system, interlinked with 200Gbit/s InfiniBand, and with NVMe SSDs for benchmark run data, and disk drives for system-level data. The TTA SPC-1 results summary table looks like this:

TTA’s SPC-1 benchmark summary table

Here’s how it compares.

With the top system scoring 7 million IOPS the TTA system doesn’t look impressive.

There are two things to consider when evaluating the TTA result. First, low-performing systems tend to have fewer configured storage drives and cost less for their performance, below $400/KIOPS, for example. They also generally run at sub-1 million IOPS. There are many SPC-1 results in this area.

Second, higher-performing systems are configured with more controllers and more drives – and cost more. But the cost-per-IOPS may be lower if they use clever software which extracts more IOPS from the hardware.

In recent SPC-1 results, three systems scored more than 1.5 million IOPS and cost less than $350/KIOPS.  These are new records for  price/performance and performance and thus relevant to customers who need that level of IOPS and  who don’t have million dollar-plus budgets.

The top performers are Huawei Fusion Storage with 4,500,392 IOPS and $329.90/KIOPS, Inspur’s AS5300G2 with 1,500,346 IOPS and $307.62/KIOPS, and, splitting the difference, TTA with 1,510,346 IOPS and $326.75/KIOPS. 

The chart below plots IOPS against $/KIOPS and shows a pack of sub-$400/sub-million IOPS entries. It also identifies higher-scoring systems and the three high-scorers with sub$350/KIOPS ratings.

SPC-IOPS and cost/IOPS ($/KIOPS)

Seen from this standpoint the TTA result is, to say the least, respectable.

There is another thing that stands out as you look at the SPC-1 listings; there are few US suppliers and it’s dominated by Asian ones, particularly Huawei.

Whatever happened to Dell EMC and HPE?

The table above shows the top 8 results; all posted since June 2017.  NetApp is the sole US vendor in the top eight SPC-1 results. An IBM system is at number 9, a DS8888 with 1,500,187 IOPS and a whopping $1,313.44/KIOPS,  filed in a benchmark run in November 2016. The next four results are from Huawei and FusionStack.

From the look of it, SPC-1 is a Dell EMC and HPE-free zone. This is curious.  Have they abandoned this benchmark? We’ll find out more.