Caringo is speeding up large video transfers with Swarm 11, an update for its video-centric object storage platform.
Swarm 11 integrates with on-demand workflows and provides:
Large file bulk upload in the content UI,
Partial File Restore (video clipping),
File sharing,
Backup to any Amazon S3 region/device.
Any Swarm domain or an entire cluster can be backed up to Amazon S3, Glacier or an S3-compliant device or service via S3 Lifecycle rules.
Multi GB-sized files are uploaded to the object store straight from a browser, using parallel ingest streams. There is no need for an intervening gateway or spooler device to ensure all file fragments are uploaded correctly.
Partial file restore – or clipping – downloads specific portions of a video, in contrast with tape video archives which transfer the entire video. This speeds editing, internal sharing and streaming. Broad CODEC support is available through an API for integration into asset management systems with MP4-based clipping available directly from the Swarm Content Management interface.
Authorised users can generate a streamable URL for any file from the Swarm Content Portal, using Swarm File Sharing. They can email, download or Slack the URL for secure internal or external sharing.
Transferring large video files to and from the cloud – a remote object store – is time consuming. CTERA addresses this with the Media Filer box, which is a local cache, and instant streaming of cloud files to reduce time to the last byte.
Caringo prefers to offer partial downloads, letting users select short clips to access in a large file. It can also upload files with parallel IO, shortening the file movement time. This is not something that CTERA offers.
Spectra Logic, the tape systems vendor, is entering the storage management market, claiming it solves the problem that most data is stored on the wrong media i.e. expensive and fast storage.
This is the world of HSM (Hierarchical Storage Management) or ILS (Information Lifecycle Management) software. But, often Spectra Logic says, old, infrequently accessed data is generally not moved off storage tiers best suited for more frequently accessed data because software products that manage this are too expensive. Typically they are priced by capacity and can cost hundred of thousands to millions of dollars.
The answer is to automate the process with affordable software and that’s what Spectra Logics’s new product StorCycle HSM is meant to do. The company said StorCycle HSM can be much cheaper than existing software and can free up a lot of costly primary storage capacity, enabling customers to avoid spending cash on no longer-needed all-flash array capacity, for example.
StorCycle classifies storage into two main tiers: primary and perpetual, and moves perpetual data to sub-tiers such as nearline disk, tape or the public cloud.
StorCycle diagram.
StorCycle can move files on a primary storage tier to the public cloud, a third-party NAS store or Black Pearl archival storage. Black Pearl is Spectra Logic’s hybrid flash/disk, front-end cache. It stores files as objects on back-end tape devices, Arctic Blue nearline disk or the public cloud.
StorCycle’s overall four-stage job is to locate and identify file data’s state, migrate old stuff, protect it and enable access to it. It runs on Windows Server and builds a file catalogue. Data can be migrated semi-manually at project directory level in a one-time event or auto-migrated continuously. Auto-migration selects file on age and/or size and is directed by settable policies.
StoreCycle media and entertainment use case.
Files can be moved in four ways:
Remove and leave nothing behind
Copy and leave original data in place
Leave symbolic link behind if file moved to a NAS target and get extended access latency
Use HTML link for files moved to tape or cloud targets (.html appended to file name in directory)
App access means using symbolic links – an approach that Komprise, a Spectra partner and now competitor, also takes. HTML access access enables the user to manually restore the target file – similar to the way files are restored from Amazon Glacier – and return it to primary storage
StorCycle has a REST API interface and supports Amazon’s S3 protocol.
Offloaded archive data can be browsed and moved back, to a target NAS say, where it can be mounted and accessed.
With files moved from primary storage, backup processes take less time.
StorCycle has four pricing tiers, starting with three admin staff and unlimited usage for $18,000, and topping out with an enterprise ‘all-you-can-eat’ license for $144,000. Support costs are 15 per cent extra. There is no capacity charge if the customer buys Spectra storage and $50/TB surcharge if they use a third-party NAS.
Spectra says StorCycle is not an HSM
Spectra puts forward the view that StorCycle is not an HSM product. It says the concepts are similar, but HSMs were invented at a time when an entire storage infrastructure was installed onsite and consisted of only disk and tape based systems.
It says HSMs are expensive systems which sit directly within an organisation’s data path. As an HSM moves data from a primary system it leaves a stub file behind in place of the original file. StorCycle doesn’t do this and it does not need to sit in the primary data path.
Blocks & Files thinks that where ‘X’ is a thing that automatically moves data between high-cost and low-cost storage media, then X meets the generic definition of an HSM product.
Dell EMC is ramping up performance for its high-end PowerMax arrays by adding support for FC-NVMe and Optane storage-class memory (SCM). Customers will get up to 50 per cent more read IOPS, lower latency and doubled throughput. PowerMax is also getting VMware, Ansible and Kubernetes integrations.
PowerMax already supports NVMe SSDs and this upgrade provides end-to-end NVMe support. FC-NVMe is the NVMe fabric protocol for storage array access implemented using Fibre Channel cabling -running at 32Gbit/s in this case. It delivers sub-200 μs latency with NVMe flash SSDs.
The updated systems support 750GB and 1.5TB dual-port Optane DC 4800X SSDs which have an NVMe interface. Dell heralded Optane support in December 2018 and PowerMax is the first system to ship with these drives.
The PowerMax’s machine learning sub-system looks at IO profiles and uses predictive analytics and pattern recognition to automatically place data on the SCM and NVMe flash SSDs.
PowerMax iterations
The first generation PowerMax array was announced in May 2018, with two models; the 4U 2000 and the rack-level 8000:
PowerMax 2000 – to 1.7m random read IOPS, 300μs latency, and 1PB effective capacity.
PowerMax 8000 – to 10m IOPS, 300μs latency, to 175GB/sec and 4PB effective capacity.
They scale up by adding PowerBrick controller/storage units; one to two for the 2000 and one to eight for the 8000. A brick includes a controller with two PowerMax directors, packaged software, cache, and 24-slot Drive Array Enclosures. The engines use Xeon CPUs; E5-2650 v4 for the 2000 and E5-2697-v4 for the 8000
The updated PowerMax specs are:
PowerMax 2000 – to 7.5m IOPS, sub-100μs read latency and 1PB effective capacity.
PowerMax 8000 – to 15m IOPS, sub-100μs read latency, 350GB/sec and 4PB effective capacity.
This is a performance upgrade, not a capacity boost.
PowerMax Competition
Infinidat claims its arrays are faster than PowerMax. A May 2019 update to its F6000 array provided up to 2m IOPS and 25GB/sec throughput. Infinidat claimed latency is typically less than 1 msec, down to under 50μs as measured from the host, in one example. PowerMax with SCM and NVMe-FC has more IOPS and equivalent latency.
HPE’s Primera array puts out 2.3m IOPS and 75GB/sec of data with sub-millisecond latency. NVMe-oF support is baked into the Primera OS but is not yet available.
Primera is also said to be ready to support dual-port Optane drives and the system can handle the IO load of NVMe-oF and SCM.
PowerMax Operations
Pre-built modules for RedHat Ansible, available on GitHub, enable customers to create Playbooks for storage provisioning, snapshots and data management workflows for automated operations.
Also available on GitHub, a Container Storage Interface (CSI) plug-in for PowerMax provisions and manages storage for Kubernetes workloads.
A VMware vRealize Orchestrator (vRO) plug-in enables customers to develop end-to-end automation routines for provisioning, data protection and host operations. These routines can be offered as self-service catalogue items on the vRealize Automation platform.
Dell EMC Cloud Storage services provide disaster recovery as a service using AWS, Azure or the the Google Cloud Platform. This supports PowerMax.
There is a Dell Technologies validated design for PowerMax; and PowerMax is validated for VMware’s Cloud Foundation through Fibre Channel as primary storage. The hardware updates (NVMe, SCM) will be available on September 16 but everything else is available now.
Cohesity has entered the copy data management market, farming out space-efficient clones of data sets to developers from its backup store.
The idea is to stop developers having to request data copies from the IT department and then neglecting to manually delete used copies that are no longer needed.
The use of copy data management (CDM) software to combat copy data sprawl was pioneered by Actifio and Delphix in 2012 and 2013. They provided virtual copies of master data for used by app development testers and others. Copies are managed and deleted when no longer needed so that storage of multiple redundant copies of data is stopped in its tracks.
Cohesity’s new software is called Agile Development and Test (ADaT) and it makes zero-cost clones (virtual copies) of database backups stored in Cohesity’s Data Platform. These can be created in a self-service fashion by authorised developers who use such virtual instances to test their application code.
Copy data is scanned to detect and mask personally identifiable information such as social security and credit card numbers. Cohesity claims the resulting smaller number of copies reduces customerexposure to security attacks and also lowers overall storage costs.
The main benefit though is on-demand access to up-to-date test data by developers. Hence the product name.
Other copy data management suppliers and offerings include;
Cohesity had an earlier CloudSpin product punting data from its Data Platform to test and dev people. We are told by Cohesity that Agile Development and Test is a very different set of capabilities to CloudSpin.
CloudSpin is aimed at hybrid cloud mobility – take an on-premises workload and spin it up in the cloud) – and the use cases are app migration to cloud, test and dev in cloud, and DR to cloud.
Agile Dev and Test is specifically focused on Database backups (Oracle, SQL Server, etc.) and seamlessly extending these backups for development and testing use cases.
ADaT is subscription-based and will be available in the fourth quarter. The self-service capability is currently beta and data masking will be available in the coming quarters.
Cisco today announced MDS 9000 storage network director support for the coming 64Gbit/s Fibre Channel interface and NVMe-FC,. It has also added Ansible automation and end-to-end analytics across ihe 64/32Gbit/s Fibre Channel portfolio.
Cisco’s MDS 9000 is a set of storage networking devices ranging from switches to directors – central very large switches or directors which aggregate switch-level traffic. The products support 16 and 32Gbit/s Fibre Channel (FC), FICON mainframe access, iSCSI and FCoE across Ethernet and FCIP (Fibre Channel over IP) protocols.
MDS 9000 product range.
Preparing for gen 7 FC
Fibre Channel speeds are expected to double in the very near future. Enabling technologies include all-flash arrays and non-volatile memory express over fabrics (NVMe-over-Fabrics) using Fibre Channel as the carrier cable.
Cisco is announcing 64G-ready capabilities on the Cisco MDS 9700 platform, with support for NVMe and through that for all-flash arrays. The company will add 64gig FC line card support to the 9700s as a non-disruptive hardware upgrade along with a software upgrade and the products will then support 16, 32 and 64gig FC simultaneously.
The upgrade from gen 6 32Gbit/ to gen 7 64Gbit/s is imminent and line card, host bus adapters and edge switches that support the new Fibre Channel standard could land by the end of the year. At that point central storage networking switches and directors will need to support the standard to provide an end-to end 64Gbit/s FC capability.
Each FC speed advance requires new hardware line cards and software added to the MDS products. Historically Cisco has been behind its main storage networking competitor Brocade, now owned by Broadcom, in introducing support for developing FC standards. It was about two years behind with 16Gbit/s FC and one year behind with 32Gbit/s FC. Now it is signalling it has caught up and will be ready with 64Gbit/s FC when the industry adopts it.
With this 9700 iteration it will be possible to run concurrent FC-SCSI and NVMe/FC workloads. Cisco believes NVMe/FC has lower hurdles for adoption than NVMe-oF using ROCE which requires data centre-class Ethernet.
Analytics and automation
Cisco has supported generic Ansible automation modules since 2017 to automate aspects of MDS operation. It is now adding specific SAN provisioning automation modules for Ansible, with VSAN, device-alias and zoning configuration automation facilities.
Ansible modules can be used to automate previous manual infrastructure procedures for provisioning compute, memory, network and storage arrays. This can reduce what previously could take hours or even days to less than a minute.
Distributed creative content teams need to share content files from a multiplicity of devices and locations. CTERA’s Media Filer enables that, adding cloud-scale and security to the mix.
CTERA produces secure cloud storage gateways cum Edge filers. It has developed this technology to build a cloud-connected edge filer box, the HC2400M, for media and entertainment-type operations.
CTERA Edge Filer product line.
This all-in-one box provides the access speed of local files, the scale of the cloud and enables media file sharing by distributed content teams. The specifications are:
Existing NAS systems can be tossed. The Media Filer provides transparent migration from them with preservation of share structure and file permissions. It also has intelligent cloud tiering which can automatically move files not needed for current projects or reference up to a cloud. This can be public or private, and S3 is supported.
Once evicted such moved files have a stub left behind, pointing to the cloud destination. This slows access and then restoration from the cloud adds to the file’s access latency. The Media Filer causes instant streaming from the cloud when a stub is opened to mitigate this.
Users can specify that certain files stay local to guarantee the fastest access performance.
CTERA has added source-based encryption, firewall fencing, virus scanning and Active Directory integration. To support distributed teams the Media Filer has:
Continuous multi-site sync between filers and endpoints
Cloud-mapped cached desktop drive for roaming team members
Native and secure access mobile app
Direct cloud upload of large files
Authenticated web access for external users.
There is built-in backup and archiving with file-level versioning for users to self-restore files.
Advertising business WPP is a flagship Media Filer customer. The Media Filer is available now through CTERA’s channel and you can download a datasheet.
All-flash array revenues fell in the second 2019 quarter and the total storage market slumped almost 10 per cent year-on-year, according to IDC
We should note IDC has altered how it calculates suppliers’ total enterprise storage systems sales in its Storage Tracker report for the second 2019 quarter.
William Blair analyst Jason Ader reports: “IDC has amended how it reports total storage numbers in August 2019 to include servers with fewer than three drives within the enterprise storage market, thus increasing the total size of the market on a historical basis.”
IDC’s number crunchers totted up total enterprise storage market revenues of $13.23bn in Q2 2018. Using the new calculation method, this more than doubles to $28.62bn. IDC’s Q2 2019 total is $25.9bn, down 9.5 per cent.
Ader says: “This compares to 4.1 per cent growth in the prior quarter and 35.9 per cent growth in the year-ago quarter.” There has been a pronounced fall in storage revenue growth, taking it into negative territory. This is attributed to cloud data centres buying fewer servers, which reduced internal/server-based storage revenue.
However this is expected to return to growth with cloud data centre storage buying from ODMs. IDC forecasts five-year CAGR of 8.9 per cent through 2023. IDC us seeing more growth in server SANs than external storage. Ader says: It expects growth in internal OEM storage systems to outpace external storage systems over the forecast period, generating CAGR of 8.5 per cent through 2023 (compared to 1.1 per cent CAGR for external storage).
Suppliers’ total storage revenue market shares in the quarter were;
Dell Technologies – 21.6 per cent
HPE (including H3C China JV) – 16.5 per cent
IBM – 6.6 per cent
Inspur – 5.7 per cent
Lenovo 4.9 per cent
The calculation change has caused NetApp and Pure Storage to fall out of the top 5 supplier ranking, to be replaced by Inspur and Lenovo.
Worldwide all-flash array market revenue for the quarter was $2.09bn, down 0.7 per cent, on the back of reduced demand and lower NAND pricing. In contrast there was 18 per cent growth last quarter and 48 per cent growth a year ago.
All-flash array supplier shares were:
Dell Technologies 28 per cent
NetApp -17 per cent
HPE/H3C -14 per cent
Pure Storage – 12 per cent
IBM – 10.1 per cent
Others – 17.3 per cent
Pure Storage is slowly but steadily growing market share;
The all-flash arrays systems market is projected to grow at a compound annual rate of 10.8 per cent through 2023.
Customers could go straight to 18TB and 20TB disk drives from current 14TB drives, bypassing the 16TB capacity level.
So says Aaron Rakers, a senior analyst at Wells Fargo, following a recent conversation with Western Digital CEO Stephen Milligan and CFO Rob Eulau. He learnt that WD has not seen any signs of significant share recapture from Seagate after its competitor launched 16TB drives. Hence the notion that 16TB could be a waystation between the14 and 18TB drive capacities.
In a note to subscribers last week, Rakers said 16TB could represent a soft cycle vs. an industry move from 14TB to 18TB.
Let’s see how Seagate’s 16TB drive shipments fare over the next two or three quarters. This will tell us if Rakers’ notion is correct.
Seagate uses 9 platters for its 16TB disk drive technology. Major customers have qualified the products and volume shipments are expected by June, 2020.
Western Digital last week announced 18TB drive technology, with a 20TB implementation using shingled magnetic recording. These also have 9 platters. Rakers estimates that WD’s cost structure is up to 20 per cent lower than Seagate can achieve for 9-platter drives, due to higher capacity and use of aluminium rather than glass platters.
These WD drives sample by the end of this year with volume ships starting in the first half of 2020. This is roughly the same time as Seagate ramps up 16TB shipments.
WD can produce a 16TB drive if demand is significant, by removing a platter or a read/write head pair from its 18TB drives.
Amazon has cut prices by 44 per cent for Elastic File System (EFS) when using Infrequent Access (IA) with Lifecycle Management.
Amazon EFS customers enable Lifecycle Management and any file not accessed after 14, 30, 60 or 90 days is automatically moved to the EFS Infrequent Access (EFS IA) storage class. This enables customers to reduce storage costs by up to 92 per cent compared to EFS Standard storage class, which costs$0.30/GB/month.
When launched in February 2019 EFS IA cost $0.045/GB/month. It is now $0.025/GB/month. “This is one of the largest price drops in the history of AWS to date!” AWS technical evangelist Steve Roberts wrote in an official AWS blog.
He cited “Industry analysts such as IDC, and our own analysis of usage patterns confirms, that around 80 per cent of data is not accessed very often. The remaining 20 per cent is in active use.”
With that in mind here’s the blog’s pricing example looking at storage costs for 100TB of file data;
20 per cent of 100TB = 20TB at $0.30/GB-month = $0.30 x 20 x 1,000 = $6,000
80 per cent of 100TB = 80TB at $0.025/GB-month = $0.025 x 80 x 1,000 = $2,000
Total for 100TB = $8,000/month or $0.08/GB-month. Remember, this price also includes (for free) multi-AZ, full elasticity, and strong file system consistency.
Total for 100TB in EFS standard = $30,000/month, meaning a saving of $22,000/month.
EFS IA with Lifecycle Management moves files from EFS to EFS IA transparently to applications. The data remains accessible within the same file system namespace albeit with a slightly higher latency; double digit ms vs single digit ms, AWS says.
Blocks & Files understands AWS views competitors as not having the capability of such a low-cost tier.
The Isilon ONeFS filer OS has been updated to v8.2.1 and adds hardware accelerated in-line compression and inline deduplication capabilities.
This version can provide up to 140 per cent greater usable capacity, depending on the workload, compared to OneFS 8.2 based solutions.
That means it can;
Scale cluster capacity up to 139 PB of usable capacity up from 58 PB in OneFS 8.2.
Deliver up to 3:1 compression and in-line deduplication on the F810 all-flash platform depending on the dataset.
In-line compression and deduplication operations on the F810 seamlessly interoperate with all existing Isilon storage platforms and all OneFS software modules.
Increased storage capacity and efficiency reduces data centre footprint and rthe esources consumed.
OneFS 8.2.1 is available now.
Portable Drives from Seagate
Seagate has announced a pair of cloth-wrapped portable drives at IFA 2019 in Berlin. The white or black cloth covered One Touch SSD holds 500GB or 1TB while the One Touch SSD Special Edition has a cover in camo patterns themed red, green, blue or white and storing 500GB.
The cloth-wrapped One Touch drives.Isn’t high-tech wonderful?
The drives include:
Seagate’s Toolkit with Sync Plus continuous backup software,
File transfer speeds up to 400MB/s sequential read/write,
Compatibility with both PC and Mac (exFAT) via USB 3.0 interface (braided USB 3.0 cable included,)
Complimentary 2-month subscription to the Adobe Creative Cloud Photography Plan and a one-year subscription to Mylio Create.,
3-year warranty.
They are available in October and MSRP prices are $104.99 (500GB) and $199.99 (1TB) for the One Touch and $109.99 (500GB) for the One Touch Special Edition.
SwiftStack
An IDC report, “SwiftStack Stakes Its Play in the AI/ML Market,” is available and says a SwiftStack reference architecture for AI/ML data management is backed by two customers. It includes Cisco servers and switches, NVIDIA’a DGX GPU servers and SwiftStack’s object storage software.
One customer uses NVIDIA DRIVE, an autonomous vehicle development platform using SwiftStack storage It supports 1OB of data per car per week with 30 cars being tested.
We’re told the SwiftStack software allows data transfer rates in excess of 100GB/sec, allowing GPU-based compute farms to stay busy.
Tsinghua Unigroup’s DRAM Fab
Chinese state enterprise Tsinghua Unigroup is building a DRAM fab, having signed an agreement with the Chongqing government to establish an R&D centre and a wafer fab for DRAM production in Chongqing’s Liangjiang New Area.
Construction is scheduled finish in 2021. TrendForce reports Tsinghua has created a DRAM business unit with Diao Shijing, former director of MIIT, as the chairman and Charles Kao as the CEO.
According to TrendForce a major problem faced by this Tsinghua effort is DRAM fab process technology. It has no external partner that can provide the necessary expertise.
Shorts
ATTO Technology is now shipping the new ATTO ThunderLink 3252 Thunderbolt 3 to 25GbitE adapter. It is optimised for extremely low latency and high-bandwidth data transfers and features the industry’s lowest power consumption.
Cloud backup and DR supplier Carbonite has a new exec team after its $619m purchase of Webroot with its machine learning capabilities. John Post is the COO. Chad Bacher is Chief product Officer, Hal Lomas is the CTO. Anthony Folger remains as CFO as does CraigStilwell as Chief Revenue Officer
Tabor Communications has a report looking at the suitability of Ceph storage SW in high-performance computing. Download it with registration here.
Chelsio announced support of the SoftiWarp feature of Linux kernel 5.3 by its line of T5 and T6 line of Unified Wire 25/40/50/100 GbitE protocol offload adapters. SoftiWarp is an open source software implementation of the iWARP protocol suite developed by IBM Zurich Research.
With it Any L2 NIC can now run the iWARP protocol and leverage Ethernet RDMA networking. Download a technical brief here.
French MSP Asema has deployed Cohesity’s secondary data converging DataPlatform for its own and customers’ workloads. Its using Cohesity to offer storage and backup services, having originally used NetApp.
FileShadow provides a thin provisioned, secure and searchable cloud vault for individuals, SMBs and SMEs on Amazon WorkSpaces virtual desktops.
Free of the lawsuits against and from Sony, Fujifilm is now producing LTO-8 tape cartridges.
An IHS Markit report says the market for storage equipment certified by the Open Compute Project Foundation (OCP) is expected to more than double from 2019 to 2023 as vendors move to offer new products compliant with the organization’s standards. Global revenue for OCP storage products will expand to $5.4 billion in 2023, up from $2.5 billion in 2019, according to the Data Center Storage Equipment Market Tracker report.
Scale-out filer biz Qumulo, which ships an Active Archive Storage system product, has joined the Active Archive Alliance. It says it has built-in intelligent read/write cache to deliver the performance needed for billions of files accessed by hundreds, or even thousands, of users.
StorONE today has become a Gold level member of Oracle PartnerNetwork (OPN). StorONE’s turnkey enterprise solution is built with Oracle Linux and Oracle VM, for DevOps environments with containers and microservices. This allows cloud-native business applications and traditional workloads to run on a single storage infrastructure.
Scale-out filer startup Qumulo is partnering with axle ai, with its AI-driven video media management SW, to provide media and entertainment customers with a system for storing, managing, and automating search of their content.
Superfast parallel file software startup WekaIO has been awarded three patents;
10268378 for “congestion mitigation in a distributed storage system,”
10394490 for “flash registry with write levelling,”
10402093 for “congestion mitigation in a multi-tiered distributed storage system.”
It has an astounding 41 more patents pending.
An ESG tech evaluation report looks at Western Digital’s ActiveScale object storage system. It’s downloadable with registration.
Digitimes reports China’s Yangtze Memory has started volume production of its 64-layer 3D NAND, fabricating 256Gbit TLC chips. The company is expected to grow its 64-layer 3D NAND flash output to 100,000-150,000 wafers monthly in 2020. It is developing a 128-layer capability.
Zstor now offers certified Azure Stack HCI solutions with the Mini Cubes as well as rack servers with the latest Xeon Scalable Gen2 CPU generation.
IDC has published a September storage tracker different from last year’s tracker, making overall enterprise storage market comparisons virtually impossible.
The tech analyst firm has put out a Worldwide Quarterly Enterprise Storage Systems Tracker that looks only at vendor revenue in the worldwide enterprise external OEM storage systems market. Last year’s quarterly tracker report covered the total enterprise storage systems market. Now we just get a look at the external enterprise OEM storage systems market.
Also this latest quarterly tracker report’s numbers for the enterprise external OEM storage systems market are different from last year’s external enterprise storage systems market. Thank you, IDC.
This IDC enterprise external OEM storage systems market report tabulates the top five vendors’ results in the second 2019 quarter:
IDC’s headline points;
Vendor revenue decreased 0.8 per cent Y/Y to $6.3bn,
Capacity shipments in the external storage systems market rose 5.2 per cent Y/Y to 16.3 EB,
Capacity shipments in the total market (including ODMs and server-based storage) declined 4.2% to 107.9 EB,
Original design manufacturers (ODMs) revenue from hyperscale datacenters declined 22.9 per cent Y/Y to $4.2bn.
Sebastian Lagana, IDC research manager, said: “Second quarter results trended similarly to the first quarter with ODMs continuing to decline against a difficult year-over-year comparison and internal (server-based) storage weighed down by a contraction in the server market.
“While the external OEM segment was pressured by contraction in all-flash arrays, which has long been a growth driver for the segment, we did note end user investment in midrange SAN platforms remained strong, with nearly all OEMs generating growth in that portion of the market.”
Looking at the vendor numbers, HPE did very well, with a 13.5 per cent rise, overtaking NetApp. Leader Dell slid a little more than the market overall, still blowing everybody else away. It has more revenue in this market than HPE, NetApp and IBM combined.
IBM lost most percentage-wise, followed by NetApp. Hitachi grew nicely, gaining ground on IBM.
Did server SANs impact external shared storage sales? Capacity-wise, apparently not. Revenue-wise, maybe a little. As noted above IDC has not released a worldwide total enterprise storage systems market report for the quarter so we are unable to assess how server-based storage is doing relative to the external storage market.
Samsung has developed an open standard prototype key:value SSD and is working with Datrium, Minio, DDN’s Nexenta and others to productise it.
A key:value (KV) SSD implements an object-like storage scheme on the drive instead of reading and writing data blocks as requested by a host server or storage array controller. In effect the drive has an OTL, an Object Translation Layer, which converts between object KV pairs and the native blocks of the SDD.
A Samsung document states: “the Samsung Key Value SSD needs only standard SSD hardware, which is augmented by special Flash Translation Layer (FTL) software that provides its processing capabilities.“
Seagate pioneered the drive level KV store idea in 2015 with its Kinetic disk drives. These had an on-drive KV store and were directly addressed by hosts as Ethernet devices. They failed to grab customers and Seagate does not appear to sell them anymore.
The SNIA has published an open standard for a Key Value Application Programming Interface (KV API) which means that KV devices will be interchangeable between suppliers – unlike the Seagate Kinetic drive.
SNIA executive director Michael Oros said in a prepared statement: “The SNIA KV API specification… paves the way for widespread industry adoption of a standardised KV API protocol.”
Samsung is making large but unquantified claims for the proto drive. It will offer substantially greater scalability, unmatched durability and CPU-relieving functionality compared to existing SSDs. However, we don’t know the proto drive’s actual capacity, flash tech, performance, interface or its endurance.
It says there are many unstructured data applications, such as Ceph, Mongo DB and Reddis. When they use software-based KV stores on SSDs, they then handle flash garbage collection (deleting old data, etc.) in the host, affecting system performance. Samsung’s KV SSD does garbage collection itself, on the drive, like a traditional SSD’s flash translation layer, so the host servers should now go faster.
The Samsung document shows RocksDB running both on a standard system and a Samsung KV SSD system, which was considerably faster in terms of queries per second (QPS) and the amount of device IO and user IO:
RocksDB performance on Samsung KV SSD
Datrium’s chief scientist Hugh Patterson is enthused: “SNIA’s KV API is enabling a new generation of architectures for shared storage that is high-performance and scalable. Cloud object stores have shown the power of KV for scaling shared storage, but they fall short for data-intensive applications demanding low latency.
“The KV API has the potential to get the server out of the way in becoming the standard-bearer for data-intensive applications, and Samsung’s KV SSD is a groundbreaking step towards this future.”
Productisation is probably a 2020 story – and possibly the second half. We’ll watch for further details as they come out.
