Intel declares “OpenEBS Mayastor is the fastest open source storage for Kubernetes,” but the documentation lacks any details that could allow comparison with other Kubernetes storage products suchs open source Ceph and Gluster or proprietary Portworx and StorageOS.
Mayastor, like Portworx, is an example of container-attached storage (CAS). The idea behind container-attached storage is that the storage controllers are orchestrated by Kubernetes – just like it orchestrates containers – as opposed to via a link to an external array. The data itself is accessed via containers, rather than an off-platform, shared storage system. MayaData’s developing open source OpenEBS CAS software runs within Kubernetes and orchestrates open source storage facilities for containers using any Kubernetes stack.
MayaData CEO Evan Powell writes: “With Container Attached Storage the default mode of deployment is hyperconverged, with data kept locally and then replicated to other hosts which has other benefits as well… The second reason that Container Attached Storage is being adopted is due to the strategic decision of many enterprises and organisations to avoid lock-in.”
Storage engines
MayaData has developed different storage engines for OpenEBS. It defines a storage engine as the data plane component of a persistent volume for a Kubernetes-orchestrated container. The cStor (container Storage) engine became available in 2018 with v0.7 of OpenEBS. It provides iSCSI block storage and supports snapshots, clones and thin provisioning. By 2019, OpenEBS recommended cStor if a cloud-native application was in production and required storage-level replication.
Mayastor was developed in 2019-2020 to take the workflow developed for cStor and make it faster, by using NVMe-oF concepts, for example. This was intended to increase IOPS capability and lower latency. Mayastor entered beta test in November and hasn’t yet been released in a production-ready state.
The CAS performance test scene is developing in fits and starts, and OpenEBS, itself in beta test, started from a trailing position in both StorageOS and Volterra testing.
Volterra testing
Jakub Pavlik of Volterra, a cloud-native platform services supplier, tested the cStor engine’s performance in Feb 2019 against Portworx, Gluster, Ceph and Azure container storage, with all of them running in the Azure public cloud.
OpenEBS cStor was the worst performer, as a chart of random read and write IO shows. It scored 23.1/2.6 MiB/sec (24.2/2.7 MB/sec) against Portworx’s 749/21.6 MiB/sec (785.4/22.6 MB/sec. It also had the lowest IOPS and longest latency.
Pavlik retested in September last year but used Mayastor instead of cStor. It provided 88.6/29.9 MiB/sec (92.3/31.4 MB/sec) random read/write throughput. He also tested Longhorn from Rancher Labs as the chart shows;
These tests are not directly comparable to the StorageOS tests but indicate that OpenEBS Mayastor would perform at a roughly similar level to Longhorn and Ceph instead of trailing far behind, as it did with the cStor backend.
StorageOS
OpenEBS’s cStor performed least well in a StorageOS benchmark test comparing Ceph, Gluster, Portworx and StorageOS.
This test used cStor and not the later, and faster, Mayastor.
Intel and Optane
A joint MayaData- Intel document describes how Inteltested Mayastor against two Optane Data Centre (DC) SSDs; the P4800X and newer P5800X, and found it contributed less than 10 per cent overhead to the P5800X’s raw device IO.
The testers measured performance in terms of thousands of queries a second (kqps) and IOPS, not bandwidth. They also changed system components during the tests, which makes it harder to compare performance with other suppliers. Intel’s main goal here was not provide a general CAS performance test result but to show how much better gen 2 Optane SSDs were compared with gen 1 Optane.
The document provides incomplete performance results.
For example, there are no kqps or IOPS numbers detailed for the P4800X system. Instead the document claims the “Optane SSD P5800X delivers 2-3 times throughput improvement over Intel Optane SSD P4800X, depending on the configuration.”
The document includes this table to indicate how Mayastor performance compares using the two Optane drives;
It uses different metrics, with undefined units and time, than the prior peak and sustained kqps and IOPS.
Summary
There is no way to generalise from this result to the Volterra and StorageOS Mayastor benchmarks. They didn’t use Optane SSDs or PCIe Gen 4, and Intel doesn’t use a bandwidth measure, all of which makes the comparison impossible. Intel’s declaration that Mayastor is the fastest open source storage for Kubernetes is fine as far as it goes, but that’s not much use in choosing between open source OpenEBS, Ceph, and Gluster, or proprietary Portworx and StorageOS on performance grounds.
Our conclusion from the three tests; Volterra, StorageOS and MayaData-Intel, is that it’s obvious that Mayastor is faster than the earlier cStor, but we don’t know by how much.
