Kioxia has launched EM6 SSDs accessed over RoCE NVMe-over-Fabric links and installed in a 24-slot EBOF — an Ethernet Bunch of Flash box — capable of pumping out 20 million random read IOPS.
Getting your head around this can be tricky. Think of a 2-rack unit Ethernet-accessed JBOF (Just a Bunch of Flash) with 12x 200Gbit/sec uplift ports to hosts which support RoCE (RDMA Over Converged Ethernet). This box is called an ES2000 and made by Foxconn subsidiary Ingrasys. Inside the ES2000 is a Marvell 98EX5630 Ethernet Switch that provides the 12 uplink ports and also connects to 24 SSDs inside the box. There is also an Intel Atom management CPU with 8GB of DDR4 memory to run its software.
The 98EX5630 connects to up to 24 hot-swap NVMe SSDs which can be in U.2 or the E3.S ruler form factor, or to 48 hot-swap NVMe SSDs in the thinner E1.S ruler format drives.
Kioxia’s EM6 drives are in the 2.5-inch format and hence will be found in 24-slot versions off the ES2000. The ES2000’s 20 million IOPS rating comes from, we think, having it filled with 48 E1.Sdrives as there will be more parallelism with 48 drives than 24. Also Kioxia’s EM6 announcement makes no mention of a 20 million IOPS capability, reinforcing that assumption.
The EM6 comes in 3.84TB or 7.658TB capacities with a one Drive Write Per Day (DWPD) endurance rating. They are native NVMe SSDs fitted with a Marvell 88SN2400 NVMe-oF SSD converter controller that provides dual-port 256Gbit/sec Ethernet access. Having 24 of them would total up to 48 links.
Kioxia says that the EM6 Series drives which support NVMe v1.4 and NVMe-oF v1.1, expose the entire SSD bandwidth to the network. The ES2000 can provide a high-availability (HA) for high-performance computing hosts because it can have multiple and redundant network connections between the hosts and the NVME SSDs. It can also provide scalable throughput by adding SSDs up to the enclosure’s maximum and then by adding enclosures.
The company also suggests that, in a high-availability disaggregated block storage expansion use case, the same configuration could accommodate multiple host systems, providing shared and scalable block storage with high bandwidth utilisation of the NVMe SSDs.
The EM6 drives EM6 Series drives have successfully undergone NVIDIA GPUDirect Storage (GDS) certification testing so they can hook up to NVIDIA A100-based systems forAI and other GPU-driven applications. Kioxia says the EM6 drives are available now through Ingrasys and doesn’t mention any other channel.
Download an ES2000 PDF datasheet here.
Comment
Kioxia has not revealed the type of flash used in the EM6 SSDs. We suspect it’s 96-layer BiCS4 generation flash organised in TLC (3bits/cell) format.
It seems to us at Blocks and Files that the two capacity options are limited and we might have expected a 15.6TB variant as well. Kioxia has just announced its CD7 SSD in the ES.3 format and it is strange that this is not available for the ES2000. Conceptually it would seem possible to stick a Marvell 88SN2400 controller front end on it, and maybe that work is going on behind the scenes.
Kioxia also produces KumoScale storage software systems which can scale to hundreds of storage nodes. These are certified servers running KumoScale software and fitted with Kioxia NVMe SSDs. The Kumoscale software supports NVMe-oF RoCE and NVMe/TCP access. Kumoscale systems are available through suppliers such as Supermicro, Tyan and Quanta.
