Blog: There is a misguided notion that key value databases are unfriendly to SSDs. That’s because unattended compaction layers on SSDs can shorten the SSD’s life.
You might think that key value code needs to be rewritten to cope with this.Wrong. I don’t claim to be an expert on key value databases, but I do know storage.
At risk of ruining the ending … just migrate cooler data off SSDs onto HDDs.
The Problem
Many key value databases employ compaction layers. If poorly administered and left un-attended, the data piles up. The compaction layers fill up again and again, and cause the data to be copied to another layer. And this cycle repeats.
The result is excessive and SSD-unfriendly write amplification; this threatens the SSD’s write endurance.
For those unfamiliar with compaction layers and Log Structured Merge Trees, there’s a great primer by Ben Stopford.
The Fix: A storage solution for very large, but cooler key value databases?
Large capacity databases store cooler, historical data and experience minimum writes. Understandably, cooler/bigger data is stored on HDDs, not SSDs. Common sense can prevail.
Understandably, performance-sensitive databases are rarely high capacity. This is for the simple reason that most admins of key value databases have commonsense and move ageing logging information to cooler, larger historical databases.
Additionally, if SSDs are used in situations where reliability is threatened by excessive writes, simply buy higher capacity SSDs. Bigger SSDs are harder to repeatedly fill up.
Compare the Terabytes Written per Day (TWD) rating of small capacity versus larger capacity SSDs to understand why this is the case. (My previous blog explains why you should buy bigger capacity drives.) Then have spares and migrate cooler data off the SSDs.
Put yourself into the shoes of a large-ish data centre which uses key value for large but low write rate (cooler) databases..
Would it make sense to capture logs for this week’s data on SSDs, and then move older log data onto cooler HDD based systems? Of course, it would.
For those few large and hot databases, does it make sense to just buy larger capacity, higher TWD SSDs for them? Of course it does.
So why rewrite key value database code when the same problem can be solved better and cheaper with good admin?
In summary, be diligent about migrating cooler key value data off SSDs and onto HDDs. For bigger, hotter databases, buy larger capacity, higher TWD SSDs, use more SSDs, and have some spare SSDs on hand for failures.
