Update; April 9, 2019 – Intel statement added as final section.
After the dust has settled from Intel’s monster gen 2 Xeon, Optane, Ethernet and FPGA launch this week the enduring question about Optane DC’s Persistent Memory endurance is closer to being settled, with a view emerging that Intel has finally delivered in Optane’s promise.
A Intel Optane DC Persistent Memory Module (PMM) | StorageReview.com – Storage Reviews gives Optane DIMM endurance at about 360 PBW (petabytes written) for the 256 GB drive.
I calculate that (360 PBW divided by 5 and then again by 365 and then by 2,546 to get to write cycles per day) to mean 1,406,520 write cycles, which agrees with MemVerge co-founder and CEO Charles Fan’s thinking.
He says 3DXP’s theoretical endurance is in the range of 1 million to 100 million writes (ten to the 6 to ten to the 8), with Intel’s Optane being towards the lower end of that.
If the Optane DIMM endures for 1,406,250 write cycles, let’s write the full drive contents every minute. How long will it last?
With the 256 GB drive we do 60 writes an hour, 1,440 a day, 525,600 a year and the drive dies after 2.675 years.
Let’s write the full drive contents every second. How long will it last then?
With the 256 GB drive we would do 60 writes a minute, 3,600 a minute, 86,400 a day, and the drive dies after 16.27 days. That looks very bad. But is it a real possibility?
Blocks & Files asked industry figures what they thought of this (simplistic) calculation.
Jim Handy
Jim Handy, the General Director at Objective Analysis, said: “My first impulse was to double-check the figures, but then I decided not to, and to instead consider whether it’s at all likely that Optane will be used in such a write-intensive environment. I would think not.
“In Memory Mode the Optane is only written to when a dirty line is evicted from the DRAM cache. It would take a LOT of cache thrashing to get even close to 60 full drive writes per hour.
“In App Direct Mode the software is choosing which items need to persist and which can be satisfactorily handled by the DRAM. I would assume that Optane’s slow writes would further motivate the application developer to limit the number of persistent writes. The only things that really need to persist are already separated out by most transactional databases because persistent stores are so slow on today’s systems.
“Charge card processing is an application that probably has one of the highest-volume requirements for persistent transactions. VISA says (as of 2017) that its VisaNet can handle 65,000 transactions per second. Let’s assume each transaction consumes 16 bytes (put your own guess here). That would take 17 minutes to fill a 128 GB DIMM, assuming that all of VisaNet used a single Optane DIMM. (A 256-byte transaction in a single-DIMM Optane system would meet your one-fill-per-minute number.)
“You may be right about the ability to wear the module out really fast, but it might not be important. Remember that we had something similar with SSDs some time back when SSD makers competed to get to 20 or 25 DWPD, perhaps even higher, before users figured out that they didn’t need that kind of endurance, especially if it cost more. Now there are many 1 DWPD SSDs on the market, even for the data centre.”
An industry expert on background
Blocks & Files was told on background by an industry expert that 360 PB written (1,406,250 complete device writes) into a 256 GB device is quite impressive, especially compared to NAND.
For comparison, a high-end NAND-based 256 GB SSD with 10 DWPD (drive writes per day) for 5 years, is only 18,250 complete device writes; 4.672 PBW. So, the Optane DIMM has approximately 77 times the endurance of an enterprise, high endurance NAND-based device.
Most typical SSDs have only 1 DWPD, in typical use, so the Optane DIMM compared to that device has ~770 times the endurance.
If Intel decides to issue DIMMs with more over-provisioning, the PBW will rise, of course.
Mark Webb
Consultant Mark Webb of MKW Ventures says the Optane DIMM is over-provisioned by 20 per cent and each die is over-provisioned by 20 per cent. He says: ‘Intel manages the endurance heavily to prevent wearout from impacting the user. This is part of the controller firmware.”
He also says Intel doesn’t specify endurance other than saying there are no limitations for the warranty period; “So you can run it full blast.”
Chris Evans
Architecting IT consultant and analyst Chris Evans said of the Optane DIMMS: “They essentially have ‘unlimited’ endurance, or at least unlimited in the normal 3-5 year replacement cycle.”
Twitter Optane endurance debate
Amnon Izhar, a Distinguished Engineer at Dell EMC, points out that it’s physically impossible to write 256 GB/sec to the DIMM. The DDR4 interface doesn’t support it.
Pure Storage Software Engineer Roland Dreier says “A DDR4 channel is 25 GB/sec, give or take,” which backs up this point. Anandtech’s Ian Cutress, supplying an Intel slide (below), says Optane is fixed at DDR4-2666, but the official numbers are around 8.3 GB/s per module for Optane (so, 50 GB/s ish with 6 per socket).”
F Steven Chalmers, an independent consultant and advisor, takes a different tack; “If we are using this as if memory, a hot kernel data structure or file directory can be rewritten to failure in minutes/hours.”
For example, an HPC; “ping-pong benchmark between a CPU and a non-coherent accelerator using memory locations could easily cycle in a microsecond, meaning a specific DRAM location could be written once per microsecond indefinitely. Such an algorithm could deliver 1.4e6 writes to a specific location in a few seconds.
“My point is not that the new tech is useless, it’s that the new persistent memory parts have to be used in a very very controlled way consistent with their write endurance, much as flash has to be used in a very controlled way. Useful tool in toolbox. Not best and highest use of persistent memory by a long shot.”
Izhar said: “if wear levelling is done right, it shouldn’t matter if user overwrites the same file/address or sprays data all across. Host/used does not access a specific physical cell – it accesses LBA or logical address.” The address that you target on the Optane DIMM will change internally over time as a result of wear levelling.
Although wear levelling is needed, the DIMM is byte-addressable and so no block erase takes place. That means garbage collection is not needed.
His verdict is that: ‘Intel has very good engineering team. It took them time to get 3DXP to market, but they delivered.”
Last word – Intel
An Intel spokesperson said: “The endurance of Intel Optane DC Persistent memory (DCPMM) for any given time frame is a function of both media write endurance AND bandwidth.
“Published PBW is estimated assuming maximum DCPMM bandwidth at the target power of 15W for 5 years across various access patterns including 100 per cent write.
“In the example chosen (256GB PMM writing full drive contents every minute or second), we are bandwidth limited, not write cycle limited. Our estimation for 5 years was 360PBW. Media write cycles would extend well past 5 years.”
