Analysis: Decentralized storage – the Web3 concept – will fail unless it deals with seven issues that are stopping its growth.
Web3 storage or dStorage (decentralized storage) is deliberately not using SAN or filer storage enclosures in a datacenter, be it on-premises or in established public clouds such as AWS, Azure or GCP. Instead, a distributed web of private or very small business IT shops contribute some of their storage capacity on a peer-to-peer basis using a shared protocol that handles the storage I/O, verifies their existence and trustworthiness using blockchain transactions, and incentivizes them via cryptocurrency payments.
Storj, with its Tardigrade dStorage, has been established and on our storage radar for three years, but there has been an upsurge in Web3 marketing recently with Flux, Filecoin, and Züs.
As these four suppliers have become familiar, so have the features and limitations of Web3 storage. There are seven aspects which mark out dStorage as distinctively different from what we would call centralized storage – which primarily refers to the big three public cloud providers: AWS, Azure and GCP. They are “centralized” in Web3 supplier terms because each of them is a very large and single supplier – the central and only supplier of storage services in their networks.
This Web3 view of them is based on the idea that a distributed web of storage suppliers with no central organization can be as fast as public cloud storage, as trustworthy and reliable, through the tech magic of blockchain and cryptocurrency. It can also be a whole lot cheaper as there are no datacenters to build and operate.
The cryptocurrency angle is intrinsic to dStorage – see Filecoin and its FILs – and its proponents view cryptocurrencies as intrinsically better than the dollars, euros, or other fiat currencies controlled by governments and subject to inflation and control. Cryptocurrency is decentralized so sidesteps the control mechanisms of fiat money. We’ll not go any further into the pros and cons of this, just pointing it out for context.
Seven pillars of Web3 wisdom
That said, the first issue holding back dStorage is cryptocurrency. Every business operates with fiat currency, and its operations are managed and regulated in those terms. Cryptocurrencies cannot be spent in the real world and need exchanging into fiat currencies – at rates that can fluctuate wildly. Cryptocurrencies have proliferated like wildfire. They are not regulated by any institutions and can be promoted and operated by fraudsters, incompetents, and dupes. Read about the FTX story to see how billions of dollars can be lost.
Web3 storage would have more chance of success if it abandoned the cryptocurrency idea, or at least became less reliant upon it.
Second, the need for blockchain processing is another thing that will hold Web3 storage back. This distributed ledger idea is computationally intensive and time-consuming. Traditional storage does not bear this computational burden.
The third friction point for Web3 storage is electricity consumption. If I store data on-premises then it flows to, for example, a storage array, through its controllers and down to the drives. A Web3 storage I/O has to be sharded, and then the pieces flow to hundreds of nodes over a potentially wide area network with blockchain transactions per node. Due to the sharding, blockchain work and WAN connectivity Web3 storage must inherently use more electricity per GB than normal storage – either on-premises in a business datacenter or in the Web2 public cloud.
Fourth, Web3 storage is slower than Web2 storage. It has no block access protocol and does not typically support SSDs for data storage, although NVMe SSDs are often recommended for blockchain processing use by the individual storage capacity providers.
Parallelizing I/O can speed up Web3 storage but the network’s latency will inherently be slower than on-premises storage. Züs, which uses a parallel I/O design, says that “while platforms like Filecoin and IPFS are good for archival storage, their slower-than-average transaction speeds present long-term limits for Web3 storage.” But Storj could be different as one user has recorded 400 to 600MB/sec upload speed and 280 to 320MB/sec download speeds.
Typically Web3 storage does not offer storage classes differentiated by latency and/or access speed. There are no service level agreements concerning access speeds or speed consistency. You get what you are given and there is no single throat to choke if you are dissatisfied. There are no tape-based archival services in the Web3 world either. Effectively it is disk storage only therefore one speed fits all.
The fifth point concerns access protocol limitations. You get file and sometimes S3 access though a wide area network. There is no block access and no Fibre Channel or InfiniBand access. NFS or CIFS or other file system protocols may well not be supported on any particular Web3 system. Applications using them would need to be modified to use Web3 storage.
The sixth problem area – and the one that makes cryptocurrency so central to the concept – is that individual suppliers need to demonstrate regularly that they are online and storing data correctly. Web3 storage needs incentive mechanisms to ensure that they will do that. Suppliers get paid in cryptocurrency for providing capacity. None of this complexity is needed in normal storage. Admins can see, constantly, that drives are connected and storage controllers are online and doing their job. Web2 storage is inherently more trustworthy in this regard than Web3 storage, where such visible online connectivity and manageability is absent.
The seventh and final point is that there are few, if any, added storage services such as snapshotting or replication or deduplication. If your workloads need them, walk away. You may find dapps (decentralized apps) in listings you can search that provide such services, but these are separate from the dStorage providers. It is a decentralized, disaggregated and disintermediated world, after all.
Who needs it, anyway?
On the one hand Web3 storage is a blockchain and cryptocurrency-based solution looking for a problem. The problem area it has alighted on is the cheap storage of backup and similar data sets that don’t need fast access. It’s cheaper than S3, but only suited for slow-access data and burdened with the computational complexity and excessive electricity use that afflicts most blockchain applications. And it’s tarnished with cryptocurrency instability issues.
Unless suppliers can persuade customers that these are soluble or even solved issues, it will have a difficult time becoming a mainstream data storage technology for enterprise and public sector use.
On the other, more starry-eyed hand, we could say that Web3 storage is Clayton Christiansen’s disruptive innovation in action. We could argue that its significantly cheaper cost will grow its popularity and capabilities, and overturn the storage world completely. To do that, it will have to sort out the seven issues we have highlighted above.
