Weka running in the public cloud has made big gains in the SPECstorage Solutions 2020 benchmark arena with clear wins in four of its five categories.
Update. Weka blog point about the withdrawal of its use of latency measures added in bootnote. 23 April 2024.
SPECstorage Solutions 2020 is a Storage Performance Evaluation Corporation (SPEC) validated test of file storage performance in five workload scenarios: AI image processing workload, representative of AI Tensorflow image processing environments; Electronic Design Automation (EDA); Genomics; Software Builds; and Video Data Acquisition (VDA). Each workload results include jobs or builds, ORT (Overall Response Time), and other measures detailed in a supplier’s submission on the benchmark’s results webpage. Weka supplied mostly winning on-premises results using Samsung SSDs in January 2022.
Weka’s VP for product marketing, Colin Gallagher, said: “What makes these results so interesting isn’t just that Weka is either raw or effectively the #1 result in all the SPEC 2020 benchmarks – it’s the impact of being able to handle any IO profile with zero tuning changes between benchmarks … For customers, this latency advantage shows up as real wall-clock time reductions in time-to-completion of jobs.”
The company’s parallel file system runs across a cluster of nodes – public cloud fast ephemeral instances – and uses kernel bypass technology, with no manual tuning needed. We have charted each benchmark workload results in a 2D space defined by a horizontal axis for the number of jobs or builds, and a vertical axis for overall response time (ORT). Suppliers with results furthest to the right complete more jobs, and then lower down complete them faster.
AI image processing
Weka ran in Azure to provide a direct comparison with Qumulo, also running in Azure, and also provided an AWS result focused on sheer speed.
Gallagher said Weka “beat Qumulo by 175 percent in raw performance at 64 percent of the infrastructure cost on Microsoft Azure. When factoring in latency, Weka can perform 2.5x the number of jobs in the same amount of time as Qumulo at 25 percent of the cost per job.”
In AWS, “Weka delivers 6x higher load count (2400) than Qumulo at 76 percent the cost per job. Net: Big or small, in multiple clouds, Weka is faster and has a better cost-per-job.”
Electronic Design Automation
Here Weka beats an eight-node NetApp AFF A900 array with a lower ORT.
Gallagher said: “NetApp: 6,300 jobs at 1.39ms ORT. Weka: 6,310 jobs at 0.87ms ORT. Weka delivered a 60 percent faster response time in the cloud vs NetApp’s fastest eight-node system (AFF A900 NVMe). Effective result: Weka can process 10,000 jobs in the same time NetApp can do 6,300.”
Genomics
Weka in the public cloud overtook UBIX Technology with 2,000 jobs achieved.
Software Builds
Here Weka did not score an outright win. “It achieved 3,500 builds with an ORT of 0.74ms, landing at #2 for load concurrency,” Gallagher said. NetApp’s eight-node AFF A900 system was faster but it “did 6,120 builds at 1.58ms ORT. Weka’s advantage of being half the latency means an effective 7,472 builds in the same time earning an effective #1 result.”
The SW Build workload in this benchmark is a continuation of the one in SPECsfs2014_SP2 and NetApp led that too with its prior AFF A800 systems. The workload description reads: “The software build type workload is a classic metadata intensive build workload. This workload was derived from analysis of software builds, and traces collected on systems in the software build arena. Conceptually, these tests are similar to running unix ‘make’ against several tens of thousands of files. The file attributes are checked (metadata operations) and if necessary, the file is read, compiled, then data is written back out to storage.”
For some reason Weka performs less well on this kind of test run, although it does its work in half the time.
Video Data Acquisition
Another clear-cut win. Gallagher said: “Weka still holds the #1 spot from two years ago with a small on-prem system (8,000 streams). We beat it with 12,000 streams. Net: On-prem or in the cloud, Weka is the highest performing video platform around.”
Weka running in AWS and once in Azure pretty much trounces the other suppliers in the survey, except for NetApp in the SW Builds workload. Overall, Gallagher concluded: “With the hyperscalers continually improving their back-end infrastructure, the public cloud can now provide comparable results to on-prem for storage.” Just not yet for SW builds, although Weka is able to claim a calculated win by taking response time into account.
Bootnote
Just to be clear, the latency points that Weka mentioned are not part of the SPEC report. An updated Wela blog removes them and notes: “In a prior version of this blog, we went beyond what is allowed in the SPECstorage 2020 reporting rules. For SPECstorage 2020, the only items to have a comparison are the load point value and ORT. Anything else, such as applying latency as an extrapolation, is an estimation and should not be used. SPEC specifically calls this out at https://www.spec.org/fairuse.html. With our sincere apologies, we have updated the blog on the recent WEKA results.”
