Pure Storage‘s International CTO thinks we’re rushing towards scalability limits in storage, networking, and software dataset size.
Alex McMullan offered Pure Storage’s stance on the scalability topic, its dimensions and issues, at a briefing in London this month. The starting point was incoming higher capacity Direct Flash Modules (DFMs) from Pure – its NAND drives, with 75 TB are shipping, a 150 TB model was previewed at its Accelerate event in Las Vegas, and a 300 TB version is on the roadmap.
McMullan said: “We’ll be making some announcements in the next couple of months.”
He thinks Samsung and other NAND suppliers are positive about reaching 500 and even 1,000 layers, and building petabyte-capacity drives.
SSD capacity is being driven higher by having more layers in a chip – Micron is at the 232-layer level, for example, and possible cell expansion from QLC (4 bits/cell) to PLC (5 bits/cell). Such higher capacity NAND drives will need more capable controllers to handle the data placement, drive wear, and garbage collection (cells with deleted data collected together and returned to use). A 150 TB drive has 150 trillion bytes to track, 1.2 quadrillion bits, and that’s without adding in an extra 10-20 percent for over-provisioning. The controller’s metadata storage and handling for this represents a major task and processing load.
“It’s much more likely that we will even retrench from QLC to go to higher numbers of layers, where you’re kind of mid-hundreds to high hundreds later this year,” McMullan added.
But he added: “There’s two different sides to this in terms of can we and then should we?”
Carbon footprints
Such silicon devices come with a relatively high embedded carbon content. “A Pure Storage array weighs about 40 to 50 kilos, depending on where it’s delivered in the world. But the CO2 footprint of that is 100 times higher. The data sheet will tell you that that same box represents 4,000 kilograms of CO2 from a manufacturing perspective – of which 92 percent is the silicon process, from cradle to grave in terms of mining a rock in a desert somewhere to crush it, to refine it, to redefine it, turn it into a wafer, and then to etch that.”
“The question is, then, if we’re heading – if you buy into the Gartner and many other analysts’ view – towards the zettabytes and then yottabyte era, can we continue to incur that same carbon cost? And what can we do about that to mitigate it?”
McMullan said Pure enterprise customers are serious about embedded carbon reduction, meaning embedded carbon in the arrays as well as ongoing electricity consumption.
“There’s a number of things we’re working on, in terms of the usual… hardware engineering, in terms of better algorithms for this and more efficiency from our suppliers.”
“We’ve foreshadowed that we’ll be building more power optimization into our platforms to run a power cap, rather than at full tilt, or to optimize with artificial intelligence to run the systems at lower power when it expects to use less… So those are all things that we’ve started already.”
“Let’s assume we continue with them in the short term. And we make a petabyte drive. I think it’s more than likely that we will hit that milestone … We’re actually working with the NAND firms [alredy] in terms of what the chip might look like, for that size drive.”
Post-NAND green drives
“Obviously, we’re in that transition from one terabit to two terabit die packaging. But we’re heading in terms of the roadmap way beyond that; four or five times, four times anyway. But finally, how we take that forward at that point is, do we continue with the paradigm?”
Meaning the NAND SSD paradigm. He mentioned a trio of alternative candidate technologies.
“I guess the three primary candidates are optical media … The second one, you’ve got the various PMems and you’ve got the MRAMs, and ReRAMs still sitting at a gigabit … that’s great for embedded systems. But we’d need thousands and thousands of chips to get even where we are today (with NAND).
“That kind of leaves you with … DNA storage, which, on the face of it, sounds very green. But given where that research direction is going, again, that involves silicon chips.
“It’s not just growing a short strand sequence and putting it in a test tube or a freezer. What they’re calling enzymatic DNA research is heading towards essentially, producing small silicon chips with little drill holes, basically you’d plug in the written medium inside each of those little holes. And then access them via silicon, which means a lot of the same embedded carbon that we already have.”
In his view, “DNA has a great story in terms of data density and can be made to work, but the current … sensor size is so painfully slow …. four bits an hour or something. Wonderful. It’s great if you want to be sending Morse code messages. But in terms of recording videos … I think it has 10 or 12 orders of magnitude to climb before it catches up with where we are today [with NAND].”
Which means that “maybe there’s a long term capability for archiving or those aspects. But realistically, that’s not a 2030 thing we think at this point in time.”
Then there is a fourth alternative: Ceramic etching on a glass substrate, like Cerabyte’s technology. McMullan has talked with Cerabyte’s US exec Stefen Hellmold.
Unlike NAND or DRAM, there is no direct electrical connection to a cell and so access time will be slower than NAND. McMullan thinks: “It’s time first byte, I think, was less than a minute. Now, maybe there’s a way around that, caching or prefetching.”
Networking
However, this is only one aspect of a multi-dimensional scaling problem. McMullan said: “We’re hearing feedback and thought processes from our customers, particularly those who are dealing with petabytes, in some cases exabytes already, and not just for AI workloads.”
“Everybody’s having the same challenges in terms of data management, data gravity, the throughput aspects that go with that, not just on the box, but also off-box.” He mentioned “Nvidia’s dominant position over InfiniBand and on Spectrum X.” That monopoly is not good, he said.
“We’ve signed up to the Ultra Ethernet Consortium as a company because we think that … Ethernet is the way forward but also that it needs to hurry up and scale … We are at 400 gig on the Ethernet side of things. We already have customers asking for 800.”
“You’ve got things like CXL, which we’re pushing on now. Specification 3.1 brings us pooling and sharing and memory. You’ve got CXL over fiber optic, which is an interesting development. And all of this is aimed at accelerating the shipping of data.”
Assuming you can store the coming massive datasets by federating systems and ship data across a network fast enough, “data scientists are telling us in terms of, we’ve got this giant cluster, we’ve got a whole massive, rapidly changing data set, which will mean indexing and tagging with rank-based technologies, huge factor databases hanging off the back of it, which are in some cases getting off at the same size as a source dataset – which was a thing that shocked a lot of people. There’s almost a double whammy when it comes to the AI tag on these things.”
He said: “I think PCIe 5 will, again, have hardware iterations on that next year. But it’s only incremental. It’s not foundational.”
This coming limit in NAND drive scaling and networking and the lack of fast enough alternatives to NAND that are also green is puzzling McMullan, along with software problems.
Software
He said: “It’s really about where we can go now in terms of where our engineers are telling us. It’s great talking about all these big drives that we’re going to make and ship. But how do we solve for a file system with 10 trillion objects in it? This is two orders of magnitude beyond what customers use today, but in the same time frame by 2030, which is where we’re pushing the engineers to go?”
“That’s what most of our focus is now, in terms of how do we build a system, which is two orders of magnitude bigger or better, or everything here that we bought today? That has impacts on compute, on memory, on networking, on bandwidth, as well as it’s actually a bigger software problem [than] hardware. We can throw more hardware at these things easily. We can go to four to eight to 12 controllers if we wanted to. How do we solve for the datasets?”
“We’re doing a lot more work on the algorithm side, on the data management data reduction side of things. … We shipped a compression card by default on bigger systems last year. There’s now a second generation of that card coming based on what we’ve seen in terms of actual telemetry, and workload profiles and cardinality.”
This is incremental, though, not foundational.
“There’s a computer science challenge with that …We support a billion files on a file system currently. And we’ve got some use cases where they’re asking for 10x. And that’s OK. But if you have to have a test system to test 200 million files in a single directory, you’ve got to have something that makes 200 billion files in the directory, which takes in some cases, days, weeks.”
“We’re now operating a scale of test where we use actual arrays as a pseudo-drive connected to another array in a test system. So we effectively have a Pure FlashArray, which is passing itself out to a bunch of other arrays and offering itself up as a drive each of these. And there’s more testing at scale currently. We’ve got a 500 terabyte drive that has been tested by us – but it’s actually an array with a pass-through connection to an appliance … That’s part of our test set up in the labs now.”
“The big problems are the carbon impact, and the software engineering to build this kind of scale. Everything else is easy. If we could magically mine pure silicon wafers out of the field, that’d be great. But it’s the impact that we’re focused on … That’s a big direction of travel for us in terms of is there a way of leveraging carbon-friendly media in our future roadmap?”
“It’s a fascinating time to be in tech. But you do feel that everywhere you go, now you’re in that maze, and there doesn’t seem to be a way out. It’s just a question of how long you can run on … We can blindly carry on and make bigger and more powerful systems based on that. But there will be a point where we have to acknowledge that the whole manufacturing chain needs to to change dramatically.”
“It’s interesting in terms of where we might go from there and lots of befuddled engineers are scratching their heads about what the 2026 platforms look like. It’s a big deal. But we’re at that point where we almost tear everything up and start again. That’s what it feels like.”
McMullan thinks he and other IT technologists in the enterprises and suppliers he talks to, who are all aware of these coming scalability issues, will likely be retired before ways of solving the problems he’s identified come along.
