The Optane Persistent Memory mystery. Can it last the course?

When Intel announced Optane memory technology in July 2015 it claimed endurance was 1,000 times greater than NAND, At the time this was taken to mean NAND endurance limitations were eliminated.

Since then the company has been markedly reluctant to discuss real world examples of Optane DIMM endurance. This raises doubts about its utility as a DRAM extender and substitute.

What does Intel say?

Blocks & Files has asked Intel three questions about this:

  • What is the write endurance of Optane DC Persistent Memory?
  • Can host server applications write direct to Optane DC Persistent Memory in app direct mode or does Intel control such write access? If so, how does it control such access?
  • If Optane DC Persistent Memory has, for example, a write endurance of 200,000 cycles, this suggests its use would be restricted to read-centric persistent memory applications and prevent its use in write-centric persistent memory applications. What are appropriate application types for using Optane DC Persistent Memory?

Intel spokesperson Simon Read sent us this note: “Our next generation Xeon Scalable processors (codename Cascade Lake) supports Optane DC persistent memory in both operating modes, memory mode and app direct (application direct) mode.

“We will be unveiling more details on Optane DC persistent memory, including read/write endurance and performance, when we formally announce the product with the general availability of our next-generation Xeon Scalable processors.”

This suggests that the November 2018 announcement of Cascade Lake AP support of Optane DIMMs was a tad premature.

Obtaining Optane

Optane is Intel and Micron’s 3D XPoint technology offering storage-class or persistent memory with near-DRAM speed and higher density but not DRAM’s near-infinite lifecycle.

Intel says Optane can function as a persistent memory adjunct to DRAM, enlarging the memory space with sub-DRAM cost, near-DRAM speed and data persistence. It says that Cascade Lake AP processors support such Optane DC Persistent Memory. However, Intel is not revealing the endurance (write cycles or cycling) of its Optane DC Persistent Memory products, the Optane DIMMs.

According to Intel documentation, the 750GB DC P4800X Optane SSD handles up to 60 DWPD (Drive Writes per Day) for five years, or 82PB written. This is is excellent compared to NAND’s 1 – 10 DWPD but not 1,000 times better.

Mark Webb, an independent semiconductor analyst, says the Intel 3DXP cycling performance “was >1M cycles… until they actually made a product. Then it started to drop.”

At 30DWPD, the P4800X Optane SSD has total write/erase cycles of 32,850 per cell, Objective Analysis analyst Jim Handy has calculated.

Compare this to Intel’s DC P3700 1.6GB standard NAND SSD, which has 31,025 write/ erase cycles per cell. This is almost identical with Optane i.e. nowhere near 1,000 times better.

Handy also calculated that a 16GB Optane DIMM has 11,406 write cycles per cell and a 32GB version 5,703. Admittedly this was in 2017 and Optane controller technology could have improved.

Direct XPoint DIMM writes

Optane DIMMs, with capacities of 128GB, 256GB or 512GB, can be written in memory mode or app-direct (DAX) mode. Intel says “!data is volatile in Memory Mode; it will not be saved in the event of power loss. Persistence is enabled in the second mode, called App Direct.”

An analyst confidentially suggested to us that Optane “DIMMs actually work with Cascade Lake SP… not just AP. The actual volumes are very low mostly due to Intel controlling it. Apparently a lot of Intel hand-holding is going on.”

He said customers want “app-direct mode for [the] fastest NVMe storage/memory. But…this is a small market (<5 per cent of Server Market TAM.)”

And he thinks Intel prevents applications writing direct to Optane DIMM memory to prevent the stuff wearing out. Is this the case?

We asked Handy about this. “Yes, applications can write directly to the persistent memory, ” he replied.”The DIMM, though, does things that are not under the control of the application program, like wear leveling and encryption, so when you ask if Intel controls write accesses I would reply that they “kind of” do, but probably not in the way that you were asking about.”

He sent us an SNIA diagram showing how persistent memory could be accessed:

Intel is an SNIA member and is, we infer, in sync with SNA views.

If applications or system software can write directly to Optane DIMMs without limit, then the DIMM could wear out, and its endurance is hugely important.

The right write cycle number

With no statement as yet from Intel there is no industry analyst consensu concerning Optane DIMM endurance.

Howard Marks, Deep StorageNet founder said: ‘I’ve seen [Optane DIMM] estimates from 50,000-500,000 cycles. I don’t really know.”

Webb told us: “If you were to cycle 3DXP with reasonable BER (Bit Error Rate) specs and DPMs (Defective Parts per Million), with its on chip over-provisioning, we believe it can be cycled about 100K times with options being implemented for 200K+. Plus I have seen third party testing showing it wears out above those cycles.”

In that case the Optane DIMM, used in a write-centric app-direct environment, could wear out in a matter of months.

According to Handy, SK Hynix recently suggested, using internal modelling, that Optane endurance is 10 million cycles, with a slide deck image showing this:

“Existing SCM” means Optane.

So we have estimates of 50,000 to 500,000, 200,000 and 10 million write cycles plus this point from Rob Peglar, president at Advanced Computation and Storage LLC: “Raw endurance doesn’t really matter, since the DIMM controller hides a lot of it. You’ve seen what they are willing to publish using 3DX inside an SSD. Do not make the mistake of assuming that’s it, though.  DIMMs are far, far different than SSDs.”

Webb also implies Optane DIMMs have controllers: “You cannot write to it like main memory and you require complex software to manage it.” That software will run in a DIMM controller.

Handy said: ” The DIMM … does things that are not under the control of the application program, like wear levelling and encryption,” which also implies a DIMM controller.

The Optane DIMM controller

If there is an Optane DIMM controller, what does it look like? We think the Optane DIMM has the equivalent of an FTL, a Flash Translation Layer. In this case it would be like an XTL, an XPoint Translation Layer. This XTL would add latency to XPoint data accesses.

intel has not clarified this point but we presume it would look after wear-levelling and over-provisioning to extend the DIMM’s endurance. That requires a logical-to-physical mapping function with logical byte, not block, addresses. With SSDs the translation layer operates at the block level. Optane DIMMs are byte-addressable so the mapping would be at byte-level.

Optane DIMMs come in 128GB, 256GB and 512GB capacities and so byte-level mapping tables contain entry numbers equivalent to the byte-level capacity. For instance, 512 x 1024 x 1024 x 1024 bytes: 1.074bn plus the over-provisioned bytes. It would need DIMM storage capacity to hold these mapping tables entries.

Roughly speaking it is as if an Optane DIMM is a DIMM-connected XPoint SSD.

Dealing with limited Optane DIMM endurance

Suppose Optane DIMM endurance is low? Handy said: “The DRAM locations that receive the most wear are things like loop counters that aren’t likely to benefit from persistence, so they are unlikely to be mapped by the system software into  persistent memory.  … We will learn more once Intel chooses to let us know.”

Marks said: ‘Because there’s still a significant latency difference between the 64GB DRAM DIMM and the 512GB XPoint one applications will create a tiered model with indices and other frequently updated data structures in DRAM and colder, though still hot by current measure, data in XPoint.”

Peglar rejects the idea that low (200,000) Optane DIMM endurance would restrict applications to read-centric use – but so what? “The entire question is baseless, without foundation, ” he said. “The assumption is entirely incorrect…3DX IS NOT NAND. Also, even if the above assumption was true (which it isn’t) – what’s wrong with read-centric memory?  It beats the hell out of no memory, having to fetch from SSD.”

And so we wait, as we have a long time already, for Intel to reveal what it knows. Let’s hope it does not disappoint with overly limited write endurance.

A final point. If Optane DIMM write endurance is inadequate for its persistent memory role this opens the door to alternative PM technologies such as Crossbar’s ReRAM and MRAM from Everspin and, perhaps, Samsung.