Home Blog Page 346

NetApp revenue decline accelerated by pandemic

By
Chris Mellor
-

NetApp’s fourth quarter ended April 24 deteriorated as the pandemic struck and exacerbated product and cloud sales weaknesses.

NetApp’s final 2020 quarter revenues were $1.4bn, dropping 11.9 per cent from last year’s $1.59bn. Net income was $196m, down 50.5 per cent from last year’s $396m.

Full fy2020 revenues of $5.41bn were NetApp’s lowest for nine years, but annual net income of $819m was the second highest in at least ten years, exceeded only by last year’s $1.17bn.

CEO George Kurian said: “The quarter started off well, and we were tracking to our targets until countries around the globe began going into lockdown. In April, the pace of business slowed significantly and our visibility was reduced.”

William Blair analyst Jason Ader summed things up: “While certain customers accelerated transactions to get ahead of the shutdown and address the demands of remote work, this dynamic was more than offset by the pace of business slowing in April, as many customers delayed large storage purchases.”

Quarterly financial summary:

  • $396m in free cash flow
  • Gross margin 66.9 per cent
  • $0.88 earnings per share compared to $1.59 a year ago
  • Cash, cash equivalents & investments – $2.88bn

Product revenues

Product revenues were $793m, with $267m from software maintenance ($242m a year ago) and $341m ($350m) from hardware maintenance and other services. These business lines accounted for about 40 per cent of revenues for the full year, and are an indication of NetApp’s massive installed customer base.

CFO Mike Berry said: “The combination of software and hardware maintenance and other services continues to be an incredibly profitable business for us, with gross margin of 83.2 per cent, which was up one point year-over-year.”

Products are divided into strategic (All-flash FAS products HW + SW, private cloud systems, enterprise software license agreements and other optional add-on software products) and mature (Hybrid FAS products, OEM products, and branded E-Series.) Both segments showed annual declines, Strategic product revenues were $544m compared to $623m a year ago, and mature product revenues fell to $249m ($377m).

The all-flash array run rate was $2.6bn, with quarterly revenue down three per cent to $656m. Susquehanna International Group analyst Mehdi Hussein talked of “four consecutive quarters of year-over-year decline in all-flash array [revenues].” It’s not good, but there is plenty of scope for sales to the existing customers as only 24 per cent of installed NetApp arrays are all-flash. 

Aaron Rakers, a Wells Fargo senior analyst, pointed out the “private cloud business (SolidFire, NetApp HCI, StorageGRID, and services) was at a $408m annual run rate in the quarter, or +19 per cent q/q. -32 per cent y/y).”

Quarterly revenue trend

What went wrong this quarter was sales close rates falling in April. But a look at a chart of quarterly revenues by fiscal year show that quarterly revenues have been falling on annual compare since Q4 fy2019; that’s five straight quarters, with the pandemic steepening the curve in this latest quarter. There’s a more systemic problem than the pandemic.

Comment

Several market forces are hitting NetApp with its traditional reliance on on-premises arrays:

  • Hyperconverged infrastructure (HCI) appliances and server SANs
  • The rise of public cloud storage
  • A move to subscription-based revenues

NetApp is adapting to each, with mixed success. Its Elements disaggregated HCI is not making a perceivable impact on an HCI dominated by Dell Technologies and Nutanix.

Public cloud-based (also subscription-based) revenues are growing fast, with cloud data services annualised recurring revenue of approximately $111m, an increase of 113 per cent year-over-year. But this is just 7.9 per cent of NetApp’s total quarterly revenue. William Blair analyst Jason Ader commented this “ramp has been well below management targets”.

NetApp is circumspect about subscriptions, with Kurian saying: “I don’t think we are trying to say that at this point in time that we are going to transition our entire business over to a subscription model. I think we need to have that in our portfolio and we’ll will have it, but we are selective about qualifying which customers we provide subscription offerings to.”

This contrasts with Nutanix, Cohesity and others ,which are moving full tilt into subscriptions.

Broadly speaking, NetApp is an on-premises, private cloud supplier in a hybrid cloud world that’s selling too little product and moving too slowly in adding public cloud and subscription-based revenues.

The company’s plan is to get in front of more buyers and more accounts and so it has hired 200 primary sales heads, focused on new customer acquisition, one quarter ahead of its previous schedule. They take three to four quarters to become productive, according to the company.

This background adds context to this month’s hire of Cesar Cernuda, previously head of Microsoft Latin America, as President in charge of sales, marketing, services, and support.

Earnings call

In the earnings call, Kurian’s outlined “two clear priorities, returning to growth with share gains in our storage business … and scaling our … Cloud Data Services business. We will exploit competitive transitions and the accelerating shift to cloud to expand the usage of our products and services.” 

George Kurian

NetApp has “created a separate globally integrated structure for our Cloud Data Services sales team with the sole focus of scaling our Cloud Services business by maximizing our partnerships with the world’s biggest public clouds.”

CFO Mike Berry said: “Our acquisition strategy will remain focused on bolstering our strategic roadmap, particularly within our Cloud Data Services business.”

Part of that business is providing virtual desktop infrastructure (VDI). Kurian said: “we are building out a modern workplace solution that combines a global file consolidation solution that we acquired in Q4 called Talon and a virtual desktop infrastructure and Desktop-as-a-Service solution provider called CloudJumper that we acquired in Q1 of fiscal year ’21.

“The combination of Cloud Volumes, Talon and CloudJumper gives us an extremely strong offering in the market, and it’s particularly suited to the broad transition of virtual desktop services to what we see as Windows Virtual Desktop.”

NetApp product leadership

Kurian is emphatic that NetApp occupies a leadership position in storage technology. “We have the leadership position, not second, not third, the leadership position in file, block, object technologies, which is the full range, and we have uniquely differentiated Cloud Data Services.” 

Yet the company is still not selling enough product. Other suppliers sell more all-flash arrays, more filers, more object storage, and more HCI systems than NetApp.

NetApp’s main competitor, Dell Technologies, this month announced the PowerStore unified, mid-range array and aims to grow its business with that product. Kurian commented: “To this particular platform, it is a long delay and still very much incomplete. And so, we see it as [a] complete opportunity for us. Now that they have announced it, we’re going after them. Stay tuned.”

Revenue guidance for next quarter is $1.17bn at the mid-point, down 5.6 per cent on a year ago. There is no guidance for the full fy2021 year; the outlook is too fuzzy.

Nutanix new customer growth and subscription push rebuffs pandemic

By
Chris Mellor
-
Nutanix logo
Nutanix logo

Nutanix latest results show it adding new customers and growing subscription revenues in the face of the Covid19 pandemic.

Sister publication The Register covered Nutanix’ third quarter fiscal 2020 earnings this moring but we have taken a look at some of the details to give you a B&F’s perspective, and they show a business that is weathering the storm.

William Blair financial analyst Jason Ader told his subscribers: “In the midst of a global pandemic, the company delivered strong billings and revenue growth, achieved record pipeline growth, and significantly reduced its operating expense run-rate. While balance sheet concerns may have not disappeared, they have certainly been alleviated as Nutanix management focuses on sales productivity and cash efficiency in this new world order, and as the company gets one step closer to reaping much of what it has sown on its subscription model transition.”

Looking at the detail, customer acquisition numbers grew by 700 in the quarter, which ran from February 1 to April 30, and now total 16,580. This is a slowdown from Q2 when 920 new customers were added and from Q1 which saw 780 new customers logged. Nutanix has a target of passing the 23,600 customer mark by fiscal 2021.

Charting 14 quarters of customer acquisition numbers shows that Q3’s 700 was the lowest total for all these quarters. 

Cash and short term investments have also taken a hit, amounting to $732.1mn at the end of Q3, down from $940.8m a year ago.  There has been a steady quarterly drop since the high point of $965.9m in Q2 fiscal ’19 as Nutanix’ growth cost it money. That’s also shown by the quarterly losses Nutanix is making;

Revenues grew 11 per cent year-on-year in Q3 ($287.6m to $318.3m), despite the pandemic.

Nutanix growth strategy has been based on two major pillars: To broaden the product portfolio beyond core hyperconverged infrastructure appliances; and to adopt cloud-style subscription billing.

It been converting business from perpetual license sales to subscriptions for several quarters and has been affected, to a small degree, by the loss of up-front license revenue. Our fourth and final chart looks at Nutanix’s quarterly revenues by fiscal year and shows a revenue growth reverse in Q3 and Q4 of fiscal ’19 and again in Q1 of fiscal 2020, but since then growth has resumed.

Nutanix has been showing revenue growth resilience in the face of the pandemic with increases in customer spend as well as having more customers. Wells Fargo senior analyst Aaron Rakers told subscribers: “Nutanix continues to highlight strong large customer traction; 910 Global 2K customers w/ cumulative spend growing 37 per cent y/y and accounts for ~30 per cent of revenue.” 

Rakers noted that “32 per cent of Nutanix’s F3Q20 deals include at least one product beyond core; company notably highlighting End User Compute/Desktop-as-a-Service strength given work-from-home demand.”

That’s a net positive benefit from the pandemic.

He also said subscription renewals are becoming more and more important to Nutanix which “reported that the renewal base of business stood at $5m in F3Q20; expected to expand to $10m. While we are in the early innings of Nutanix becoming a subscription renewal story, we think Nutanix’s commentary and incremental disclosures will begin to drive this as an increasing narrative / future positive driver of top-line revenue and, more importantly, operating leverage.”

Ader explained this reasoning: “[Nutanix] management noted that more than 90 per cent of quarterly bookings today are composed of new and up-sell business, which dampens visibility and carries high sales and marketing costs.

“Once the company begins to see a meaningful renewal pool (expected to be only $10 million in the fourth quarter) and aligns sales incentives properly, it should not only achieve better visibility but also see significant operating leverage (as renewal deals are transacted at much lower cost than new or up-sell deals). This to us is the essence of the bull case on Nutanix going forward.”

He added: “Our field checks point to hyperconverged infrastructure emerging as a clear winner in the post-COVID-19 world as customers rethink their long-term infrastructure architectures with a focus on cloud-like simplicity, automated operations, easy setup, built-in burstability, and disaster recovery. Last but not least, we view Nutanix as a highly strategic asset with the potential for takeout interest from larger players that desire a best-of-breed private cloud/HCI platform.”

Comment

What is the big story here, the main thrust?

There are just two dominant hyperconverged infrastructure appliance vendors in the market; Dell Technologies and Nutanix. Nutanix’ growth is simply not being stopped by Dell Technologies’ VSAN and VxRail products. This billion dollar run rate SW-defined server application platform hybrid cloud company is set for the long-term.

Dell’s OneFS for Google Cloud. What do the competitors think?

By
Chris Mellor
-

Last week, Dell Technologies announced Isilon scale-out filers in Google Cloud, which it has branded as OneFS for Google Cloud.

OneFS for Google Cloud is not a cloud-native offering. Netapp’s Azure NetApp Files, with ONTAP arrays running in Azure data centres, is a better comparison with Isilon hardware and its OneFs software running in GCP data centres.

With its customer-dedicated hardware, OneFS for Google Cloud is akin to a managed tenant offering, as NetApp’s John Rollason, senior director, global revenue marketing, tweeted: “not a Cloud service as such, but more a (quite fast) traditional single tenant managed service.”

Qumulo also offers its scale-out filesystem on Google Cloud Platform and as such can be considered a competitor for OneFS.

We asked Qumulo product director Molly Presley for her take on the Dell’s new service.

Blocks & Files: Dell has announced OneFS for Google Cloud. Does Qumulo have general views on this?

Molly Presley: Dell announced this relationship two years ago (2018) at EMCWorld as an early access program. They now are saying it is available in three countries. Essentially it is the same architecture that Isilon appliances have but co-located in the GCP data centers.

Blocks & Files: What do you see as the OneFS for Google Cloud advantages and disadvantages?

Molly Presley: The advantage is proximity to the cloud and a new more cloud-like consumption model. The fact that it took two years from the original announcement to get to this point would make one question how much adoption they have or why it has taken this long.

A disadvantage is that it appears this is simply Isilon appliances co-located or hosted in the GCP (Google Cloud Platform) data centers. Each customer is assigned their own hardware cluster. So this is similar to the NetApp/Azure relationship. They do not get the advantage of the economics, scale or speed of the massive GCP cloud architecture. And, customers will still suffer from the same architectural limitations inherent in Isilon’s file system (endless tree walks to do tiering jobs, no real time data visibility, inefficient small file storage).

Blocks & Files: An ESG study reported 200GB/sec aggregate read throughput and 121GB/sec aggregate write throughput for OneFS for Google Cloud. What do you think of these performance numbers [We identified Vendor X in the ESG document as NetApp and its Cloud Volumes for GCP. This provides cloud-native file services, such as NFS and CIFS, in GCP.]

Molly Presley: I asked our data scientist team to comment about the performance tests to see what kind of apples-to-apples we can provide for a Qumulo running cloud natively vs. the architecture Isilon tested.

Related to the performance paper from ESG. It seems ESG’s methodology to compare a 2PB all-flash array to a small configuration of a NAS (no matter who Vendor X is) is flawed.  I don’t understand how they would see that as a relevant comparison for a benchmark or why it would be included in an otherwise quite solid write up.  I also do not know who vendor x is.  

However, creating a Qumulo configuration similar to the one benchmarked by ESG for Isilon, we would produce the following performance; 450GB/sec read throughput and 330GB/sec write throughput.

This is a good apples-to-apples comparison.  

Comment

Isilon users get a single environment spanning on-premises deployments and Google Cloud with GCP compute able to process OneFS for Google Cloud-held data. That’s good news for those Isilon customers who want to burst compute to the public cloud. It also provides a potential platform for Isilon disaster recovery in Google Cloud.

Blocks & Files expects Dell EMC to add Power-style branding to Isilon, following the announcement of PowerStore as a unifying SC, XtremIO. Unity and VNX brand. Perhaps PowerScale is on the list.

We also expect OneFS to develop further to provide a better fit to the needs of a hybridising cloud and containerising enterprise IT environment. A porting of the software to Google Cloud or another public cloud would be in line with this.

Pavilion Data adds S3 object storage to parallel all-flash array

By
Chris Mellor
-

Pavilion Data has fulfilled a road map promise to add S3-protocol object storage to the RF100 multi-controller all-flash array

At launch last month the array – which Pavilion dubs a “Hyperparallel Flash Array” (HFA) – supported 20 controllers in the 4U enclosure. The controllers talk to NVMe SSDs across an internal PCIe network and the array delivers 120GB/sec read, and 90GB/sec write bandwidth, with 20 million IOPS at 40µsec latency. It supports access via NVMe-oF/TCP, NVMe-oF/ RoCE, NFS, iSCSI, Ethernet and InfiniBand.

Blocks & Files diagram showing Pavilion Data multi-controller scheme. Note 20 controllers are supported.

Gurpreet Singh, Pavilion CEO, said in prepared statement yesterday: “With the addition of native fast object support to the Pavilion HFA, customers can, for the first time, enjoy both industry leading high performance and ultra-low latency from one platform in a small footprint.”

The S3 storage adds to the existing block and file access and enables tiering to the public cloud for longer term storage.

Pavilion Data hardware board.

Performance

Pavilion said HFA S3 storage delivers 80 GB/sec read and 35 GB/sec write throughput. This is much faster than most object stores, certainly disk-based and dual-controller ones, but MinIO and OpenIO have both demonstrated greater speed.

OpenIO achieved 1.372 Tbit/s throughput (171.5GB/sec), using an object storage grid running on 350 commodity servers. MinIO exceeded that, delivering 183.2 GB/sec (1.46 Tbit/s) on reads and 171.3 GB/sec (1.37 Tbit/s) on writes. The company thinks it can achieve 2.92Tbit/s read bandwidth – 366GB/sec.

HW-boosted MinIO

Pavilion is using a derivative of MinIO object software technology, “optimized to leverage our IP of our overall Hyperparallel Storage Solution,” director of Product Marketing Keith Parker told us.

The Minio system above used a 32-node set up. Parker told us: “Pavilion would need to offer only 3 Pavilion HFA arrays consuming 12U of rack space, which would give Pavilion 240Gb/sec read.” 

He says Pavilion is also faster than MinIO in AWS when compared on a per-rack unit basis: “A testament to our integration/implementation of MinIO is the performance we get Per RU relative to MinIO’s published numbers on AWS.

“If you normalise this to single Rack Unit (assuming the AWS servers they are using are 2U) Pavilion is; 7x the read performance, 3.3x the write performance [and] 8x the PB capacity. This is all using  “the same software”,  so clearly our implementation and integration to our hardware makes a material difference.”

In other words MinIO SW is fast but Pavilion HW makes it go faster still.

Phased namespace extension

The initial Pavilion S3 implementation is limited to one S3 namespace per controller. Phase 2 will see a single S3 namespace across multiple controllers,  and phase 3 extends that to a single namespace across multiple HFA systems.

The S3 data can be accessed via S3, only not by any file interface. Other object storage suppliers, such as Cloudian and Scality, have added file access to their object storage.

Will NVMe become the universal block storage access protocol?

By
Chris Mellor
-

Facebook’s virtual OCP summit earlier this month hosted two provocative presentations that suggested NVMe could become a universal block access protocol. How realistic is this prospect?

The 2020 OCP Virtual Summit, held May 12 to 15, featured updates on the OCP storage project and speakers who floated NVMe as a replacement for SATA and SAS interfaces for solid-state, disk and even tape drives.

Currently, data sent to or read from disk and tape drives passes from the host’s DRAM across a PCIe bus to a SAS or SATA interface and thence to the drive. A conversion process is required when the data passes from the PCIe realm into the SAS or SATA realms.

SAS and SATA

SAS – Serial Attached SCSI – was developed to access disk and tape drives using the SCSI command set. Features include error recovery, reservations and block reclamation SAS is full duplex – it can be used to read and write simultaneously.

Lower cost SATA – Serial Advanced Technology Attachment – is half-duplex – you either read or you write but can’t do both at once. This was developed from the Parallel ATA standard to interface disk, tape and solid state drives to hosts.

SATA was announced in 2000 and has developed alongside SAS, which made its debut in 2004. SAS is considered more of an enterprise system protocol and is preferred for servers, while SATA is used, for example, to connect disk and solid state drives in a PC.

NVMe

SAS and SATA are serial access methods whereas Non-Volatile Memory Express (NVMe) is parallel and can read and write data in parallel streams to and from an SSD’s NAND chips. It is faster than SAS and SATA and can stream more data. NVMe provides faster data access for SSDs directly attached to a host systems’s PCIe bus.

A PCIe switch could be used to link many drives to the bus, just as an SAS expander is used to link disk drives to a SAS interface.

SAS, SATA and NVMe are all block access protocols and so can be used for block access devices such as tape, disk and solid state drives. However, disk and tape drives are serial access devices, unlike SSDs, and therefore providing NVMe access is a waste of protocol bandwidth.

That said, a universal NVMe drive interface would simplify system design, with no need for SAS and SATA protocol conversion when their data streams cross into the PCIe bus.

OCP Storage Project sessions

Jason Adrian, a project lead at The OCP Storage Project, and in his day job a storage hardware architect at Microsoft Azure, presented a session called “NVMe HDDs – the future starts now”.

In his presentation abstract, he writes: “The SAS and SATA interfaces have dominated the storage market for nearly two decades, but is this the right interface for the future? For SSDs, the market is already shifting to NVMe for its higher bandwidth, lower latency, and overall light-weight interface. For the HDD market, SAS and SATA are plenty fast for many more years. Why would we consider NVMe? Join us as we explore some of the un-tapped opportunities with NVMe HDDs.”

Matt Shumway and Mohamad El-Batal, from Seagate’s Office of the CTO, presented a session called “Can Block Storage Devices Converge on NVMe Protocol?”

They write: “In the last ~40 years, some of us have seen many evolutions of block storage protocols come and go: IDE/ATA, SCSI, PATA, P-SCSI, SATA, USB, FCP, SAS … NVMe. Each protocol optimised specific Enterprise or Consumer storage device solutions and features. Regardless of the various storage media types and characteristics, is it finally possible for the storage industry to unify and consolidate around NVMe as the optimal block storage protocol? … Why? … When?”

Here we have two sessions, teasing attendees with the prospect of a universal NVMe block access protocol. The presentation slides are not yet generally available. However a Seagate slide has appeared in a Forbes article by analyst Tom Coughlin:

The slide text notes NVMe would be good for multi-actuator disk drives, which contain two sets of read/write heads.

NVMe-accessed disk drives could also enable composable systems, built using the PCIe bus, to include disk drive storage alongside other elements – CPUs, GPUs, memory, SSDs, network switches and so forth.

Infinidat today announced NVMe/TCP support for InfiniBox arrays, which store data on disk drives. So, this software update delivers NVMe access to disk drive storage – we think this is an industry first. According to Infinidat CTO Brian Carmody, NVMe will be the pervasive block access method for multi-petabyte enterprises by 2025.

Infinidat ‘pours NVMe-oF rocket fuel’ on DRAM caching fire

By
Chris Mellor
-

Infinidat has added NVMe-oF support to its storage array operating system, in a move intended to preserve its speed advantage over all-flash rivals.

Today’s software update marks the first time a mainstream storage array vendor has offered NVMe access to disk drive-stored data, albeit with a memory cache sitting in front of the disks. Infinidat thinks NVMe will be the pervasive storage access protocol in multi-petabyte enterprises by 2025.

Infinibox 5.6, the latest array operating system release, adds NVMe-over-Fabrics (NVMe-oF) support and extended replication. Brian Carmody, Infinidat CTO, told Blocks & Files: “NVMe-oF with RAM-based IO is like pouring rocket fuel on a fire.”

Infinidat arrays store bulk data on nearline disk drives and rely on memory caching to speed data access times faster than all-flash arrays can serve their data from SSDs.

The new software will be generally available by July 31 and all Infinidat gen 3 arrays are ready for NVMe-oF. They require a non-disruptive software upgrade.

Infinidat today announced the joint 5.5 and 5.6 release of InfiniBox software. 5.5 adds three-way replication to enable a disaster recovery (DR) arrangement. This is not limited by latency or distance, which means InfiniBox DR extends from metro to geographic distances.

The company has added NVMe-oF across TCP/IP – but not Fibre Channel (FC) – with Infinibox 5.6. It says customers are already pleased with the performance of Fibre Channel access InfiniBox arrays, hence its decision to implement NVMe-oF across TCP first.

The NVMe/TCP facility cuts Ethernet link transfer time from about 100 microseconds to around 30 microseconds.

Infinidat slide – iPhone photo from Zoom briefing.

NVMe-oF using RoCE is faster than NVMe/TCP. A year ago, Infinidat showed the round trip latency to an Infinibox array using iSCSI was 172μs for reads and 188μs for writes. This reduced to 32μs for reads and 38μs for writes when using NVMe-oF RoCE.

Infinidat will extend NVMe-oF support for Fibre Channel and RoCE, when customer demand dictates. There are no committed dates yet. Blocks & Files speculates that NVMe/FC will appear before RoCE as this will enable Infinidat to offer NVMe-oF to 95 per cent of the installed base.

Many data centres are choosing to focus on Ethernet rather than Fibre Channel, according to Infinidat. NVMe/TCP adds faster access, at RDMA speeds, to existing iSCSI block access across general Ethernet. NVMe-oF using RoCE (RDMA over Converged Ethernet) requires more expensive data centre-class lossless Ethernet.

Price changes

Infinidat has introduced a new price structure, which includes no-charge migration, 100 per cent availability down to single array level and elastic pricing for burst capacity.

Infinidat has made deployment easier by offering free data migration from existing arrays for new customers. It claims customers migrating from all-flash arrays can double their capacity at minimum without increasing their budget. They should also get a data access speed increase when migrating from all-flash arrays, as memory caching is faster than the SSD access speeds of all-flash arrays.

The company has extended its 100 per cent data availability guarantee to cover single InfiniBox arrays from the minimum of two arrays necessary up until now. The guarantee covers any downtime caused by Infinidat.

An Infinidat customer, Igal Janni, CIO of IFOM (FIRC Institute of Molecular Oncology) in Italy, provides a quote: “Infinidat guaranteed us data availability and they have delivered – we’ve been running InfiniBox for two years and haven’t experienced a single second of downtime.”

The Elastic Pricing scheme lets customers buy a base capacity amount of storage and rent burst capacity when required. Burst capacity can be converted to base capacity without additional fees or penalties. Elastic Pricing updates Infinidat’s existing capacity-on-demand deal.

Are DPUs ready for the data centre mainstream?

By
Chris Mellor
-

The X86 CPU is a general purpose processing marvel but it is poor at handling the repetitive processing of billions of instructions. Dedicated parallelised hardware such as Nvidia’s GPUs accelerate these operations.

Other suppliers are pursuing this specialised processing hardware idea with dedicated storage and networking processors called Data Processing Units (DPU).

As the volume of data stored and sent over networks by enterprises carries on rising, DPU developers are betting that their products will be needed to accelerate storage and networking functions carried out by industry standard servers. These DPUs can be based on different kinds of processing hardware – for example, ASICs, FPGAs and SoCS with customised Arm CPUs. 

In a recent blog, Kevin Deierling, an SVP at Mellanox, a Nvidia subsidiary, argues that the DPU has “become the third member of the data centric accelerated computing model [where] the CPU is for general purpose computing, the GPU is for accelerated computing and the DPU, which moves data around the data centre, does data processing”.

Pensando DPUs deployed in servers.

A DPU is a system on a chip (SoC), containing three elements, according to Deierling.

  • Multi-core CPU, e.g. Arm
  • Network interface capable of parsing, processing and transferring data at line rate – or at the speed of the rest of the network – to GPUs and CPUs,
  • Set of acceleration engines that offload and improve applications performance for AI and machine learning, security, telecommunications, and storage, among others.

Dieirling reveals his Mellanox bias with this statement: “The DPU can be used as a stand-alone embedded processor, but it’s more often incorporated into a SmartNic, a network interface controller,” such as Mellanox products.  A SmartNIC carries out network traffic data processing, unlike a standard NIC where the CPU handles much of the workload. So, Deierling is making the point here that DPUs offload a host X86 CPU.

Arm DPU ideas

Several storage-focused DPU products using Arm chips have recently arrived on the scene. Wells Fargo senior analyst Aaron Rakers explained Arm’s DPU ideas to subscribers. He says Chris Bergey, Arm SVP for infrastructure, “shares Nvidia’s vision of the datacenter as a single compute engine (CPU + GPU + DPU). He noted the company is focused on offloading data from the processor via SmartNICs and on the use of data lakes. We continue to see an Nvidia Arm CPU in the datacenter as an intriguing possibility.”

He adds: “As Nvidia looks to integrate / leverage Mellanox, a long-time user of Arm, we question whether Nvidia could be on a path to further deepen / develop their use of Arm for datacenter solutions going forward.”

Let’s have a look at some of the DPU players, starting with AWS.

AWS Nitro

Amazon Web Services has decomposed traditional servers, adding Nitro IO acceleration cards (ASICs) to handle VPC (Virtual Private Cloud), EBS, Instance Storage, security and more with an overall Nitro Card Controller.

An AWS Elastic Compute Cloud instance is based on PCIe-connected Nitro cards plus X86 or Arm processors and DRAM. There are various EC2 instance types – either general purpose or optimised for compute, memory, storage, machine learning and scale-out use cases.

AWS Nitro architecture slide

Diamanti

Diamanti is developing bare-metal hyperconverged server hardware to run Kubernetes-orchestrated containers more efficiently that standard servers. Diamanti D10, D20 and G20 servers are fitted with Ultima-branded network and storage offload cards – DPUs with PCIe interfaces.

Diamanti D10 server’s offload cards.

They offload storage and networking traffic from the server’s Xeon CPUs to their dedicated processors, which have hardware-based queues.

The network card has a 40GbitE QSFP controller and is Kubernetes CNI and SR-IOV-enabled. It links the server to a switched infrastructure. The controller can integrate with standard enterprise data centre networking topologies, delivering overlay and non-overlay network interfaces, static and dynamic IP assignment and support for multiple interfaces per Kubernetes pod.

The storage card links the server to an NVMe storage infrastructure. It is an NVMe controller and Kubernetes CSI-enabled with data protection and disaster recovery features such mirroring, snapshots, multi-cluster asynchronous replication, and backup and recovery. Replication is based on changed snapshot-blocks to reduce network traffic.

Both cards have Quality-of-Service features to ensure storage IOPS and network throughput consistency.

Fungible Inc

Startup Fungible Inc is developing a DPU that can compose server components and accelerate server IO.

Fungible DPU diagram.

According to Fungible, network and DPU devices can perform data-centric work better than general-purpose server CPUs. Dedicate them to application processing and have the DPUs look after the storage, security and network IO grunt work.

This startup is expected to formally launch product later this year.

Pensando

Pensando makes dedicated customisable P4 Capri processors which are located at a server’s edge; the place where it talks to networks and storage. They run a software stack delivering network, storage, and security services at cloud scale with claimed minimal latency, jitter and low power requirements. 

A Pensando Distributed Services Card (DSC-100) is installed in a host server to deliver the Capri processor hardware and software functions. This DSC-100 effectively combines the functionality of Diamanti’s separate network and storage cards.

Comment

None of the DPUs above can be classed as industry-standard. Unless included in a server or storage array on an OEM basis, they require separate purchase, management and support arrangements and add complexity. Therefore, the best scenario for enterprise customers is for the mainstream suppliers to treat DPUs as just another component to incorporate into their own systems. Also, we can think about Intel’s views here. The chipmaker already makes Ethernet switch silicon and Altera FPGAs and may be keen to protect its data centre silicon footprint from DPU interlopers.

Hyperscaler customers are a different story as they have the scale to buy direct from DPU suppliers and get DPUs customised for their needs. AWS Nitro ASICs are a classic example of this.

Your occasional storage digest including Arcserve, Excelero and WekaIO

By
Chris Mellor
-

There’s a lot of teamwork going on in data storage land as suppliers piggyback on each other’s strengths. We see it with Arcserve and Sophos, WekaIO and Valohai, and Zadara and Cyxtra. You’ll also notice an AI flavour in today’s news roundup. Let’s start with Arcserve and Sophos.

Arcserve arm-in-arm with Sophos for over-long product names

Data protector Arcserve has teamed up with cybersecurity software vendor Sophos to protect backups from cyberattacks and data loss,.

The idea is to combine anti-ransomware and other threat prevention technologies with immutable backup and disaster recovery (DR) capabilities for protection from cyberattacks, major disasters, human error, or other unplanned outages.

Arcserve is introducing cyber and data protection for infrastructures with on-premises, cloud, and SaaS-based workloads. Its new cloud and SaaS offerings include Arcserve Cloud Backup for Office 365 Secured by Sophos and Arcserve Unified Data Protection (UDP) Cloud Hybrid Secured by Sophos. These are possibly the longest product names that Blocks & Files has encountered. Even the acronyms are a mouthful – ACBO365SS and AUDPCHSS.

ACBO365SS provides cloud-to-cloud backup for Exchange Online, OneDrive for Business, and SharePoint Online. It includes cyber protection, policy-based management, restoration to Office 365, and granular recovery with egress included.

AUDPCHSS is offered as a fully managed service extension to Arcserve UDP software and appliances, it provides cyber protection, policy-based management, RPO and SLA validation, application-level recovery, and failover and failback to public and private clouds with egress included

The Sophos technology is said to be AI-powered with deep learning capabilities that detects both known and unknown malware without relying on signatures, exploit prevention, and anti-ransomware capabilities.

Excelero gets involved with AI

PNY Technologies has launched the 3S-2400 AI storage server, which uses Excelero NVMesh NVMe-over Fabrics software to feed data to an Nvidia DGX-1 GPU server.

AI storage systems architect Mark Klarzynski said in a quote issued by PNY: “AI does not need snapshot or deduplication or many of the features standard office-focused storage require. It needs ultra-low latency and tremendous bandwidth at a price that does not impact the investment into GPU’s.”

PNY 3S-2400 AI storage server.

The 2U 3S-2400 has gen 2 Xeon SP (Cascade Lake SP) CPUs. It is hooked up to a DGX-1 and its 8 GPUs across a dual 100Gbit Ethernet or InfiniBand link. These access the 3S-2400’s NVMe SSDs directly, using RDMA facilitated by Excelero’s NVMesh software. The 3S-2400 has 32TB to 360TB of NVMe flash capacity filling its 24 drive slots. GPU access latency is about 90µs.

Each 3S-2400 supports 23GB/sec bandwidth and more can be added to scale-out a system.

Excelero CEO Lior Gal says the system “delivers the highest performance per rack unit in the market”.

WekaIO gets involved in Weka AI

WekaIO, the high performance storage company, is partnering with Valohai and its deep learning pipeline management system. This is part of WekaIO’s Weka AI ecosystem with reference architectures and software development kits to aid the use of WekaIO’s software in AI applications.

The aim is to run the entire data pipeline’s workflow, from ingesting data, to batch feature extraction, hyperparameter optimisation, inferencing and versioning, on the same production-ready storage platform, either on-premises or in the public cloud. A reference architecture exists for Nvidia’s DGX- 1GPU server.

Shailesh Manjrekar, head of AI and strategic alliances at Weka, issued a quote: “Our partnership with Valohai and our integration with its Deep Learning Pipeline Management tools expand Weka AI’s capabilities to offer Explainable AI (XAI). This is a critical factor for use cases with a social impact, including autonomous driving, healthcare, and genomics.”

Valohai DLMS integrates with WekaFS, which leverages i3.n Amazon Elastic Compute Cloud (Amazon EC2) instances with Non-Volatile Memory Express (NVMe) flash for the performance tier and extends the file namespace over Amazon Simple Storage Service (Amazon S3) buckets for the capacity tier.

Eero Laaksonen, Valohai’s CEO, said the WekaIO partnership “helps businesses to build models 10x faster, regain 35 per cent of lost cloud costs, and free their DataOps teams with automated ML orchestration, data management, and data mobility. It is a win-win.”

The tools from Valohai are supported in an Amazon Virtual Private Cloud (Amazon VPC), and available in AWS Marketplace. Weka IO is an Advanced Technology Partner in the Amazon Web Services (AWS) Partner Network.

Shorts

Software-defined storage (SDS) supplier DataCore has published a sponsored report that shows about half of the enterprise storage market is looking towards SDS or hyperconverged infrastructure (HCI) to meet primary and secondary storage needs. The report authors says enterprise IT increasingly recognises that HCI is primarily a subset of SDS – in other words, it is a way of implementing SDS.

Container storage startup Diamanti is expanding operations in Latin America and EMEA, and id forming partnerships with channel partnerships in Asia Pacific. It has opened an office in Amsterdam, Netherlands and has hired a team of experts with experience in Kubernetes in the UK, Benelux/Nordics, and Turkey/Middle East regions. Diamanti has also opened a Mexico City office.

Kaseya has launched a unified backup portal for Unitrends and Spanning. The Unitrends Unified Backup Portal enables customers to manage all backups from a single portal and initiate appliance, direct to cloud, and Microsoft O365 application backups.

Spanning Dark Web Monitoring is a first-to-market monitoring facility for MS Office 365 and, next month, GSuite. The software combines backup and restore functionality for Office 365 and GSuite with Dark Web intelligence and search capabilities to identify, analyse and proactively monitor for an organisation’s compromised or stolen employee data.

HPE has introduced storage-class support with NimbleOS v5.2. An all-flash AF60 or AF80 array can have 1.5TB Optane SSDs added with the OS caching technology, ensuring 95 per cent or more reads are from the Optane cache.

Cloud file gateway and sharer Nasuni is offering free migration for Panzura customers, following Panzura’s purchase by private equity.

Netlist has introduced its NS1554 NVMe M.2 format SSD. It uses 96-layer TLC NAND and delivers up to 3.0GB/sec sequential read throughput and 440,000 random read IOPS. The drive’s over-provisioning can be customised to support more than 3 drive writes a day for the 5-year warranty period. It also has flexible power throttling and power-loss protection. Capacity levels range from 800GB to 3.84TB. 

BIOS IT will sell Panasas HPC storage throughout the US, EMEA and APAC as an appointed value-added reseller (VAR), providing customised HPC product and services to its customers. BIOS IT general manager Ian Mellett comments, “We are excited to partner with Panasas to offer our customers in the HPC arena the extreme performance, reliability, and simplified manageability that the ActiveStor Ultra product delivers.”

NVMe-oF storage supplier Pavilion has launched HPCinnovation.com, a website designed to bring awareness to a “third wave” of storage for High Performance Computing (HPC) environments. It says this third wave is facilitated by NVMe which freed SSDs from legacy disk interfaces. Pavilion’s Hyperparallel Flash Arrays eliminate the bottlenecks imposed by the legacy all-flash array dual-controller architecture with a switch-like design, This enables multiple controllers to access storage simultaneously and  deliver “unprecedented performance”. 

In-memory database supplier Redis has announced RedisAI. It reduces the time spent outside the AI inference engine to collect and prepare the necessary reference data. Redis says it can and can deliver up to ten-times more inferences than other AI serving platforms and at a much lower latency. RedisAI has built-in support for major AI backend systems (TensorFlow, PyTorch, and ONNX Runtime), and enables inferencing to run across platforms. For instance, a model that was trained by PyTorch can be inferenced by TensorFlow.

Seagate has signed an Open COVID Pledge and granted free access to all of its patented technologies to help enable the diagnosing, preventing, containing, and treating of COVID-19.

The SNIA has a live webcast coming on June 23 at 10am PT, entitled “A Multi-tenant Multi-cluster Kubernetes Datapocalypse is Coming.” Sounds dramatic. It asks what are the implications of multiple Kubernetes clusters in a broad deployment? What happens if a cluster goes down? What’s the impact on business resiliency? Managing and securing multiple clusters is becoming a key topic and area of debate. Multi-cluster Kubernetes that provides robustness & resilience is rapidly moving from “best practice” to a “must have”.

StorJ Labs has released Storage Node for QNAP Systems, allowing users to generate revenue from their unused NAS QNAP hard drive capacity and bandwidth. Users can install the app and set up a Storage Node on the Storj Network in minutes. The company has also announced early access for the Tardigrade S3 Gateway for QNAP, which allows NAS owners to backup the data on their devices to the Tardigrade Decentralized Cloud Storage Service through QNAP’s Hybrid Backup Sync (HBS 3) tool.

Western Digital has joined the the Automotive Edge Computing Consortium (AECC), a cross-industry group working to drive best practices for the coming vehicle, computing and storage convergence.

Hybrid cloud data warehouse supplier Yellowbrick Data is partnering analytics supplier SAS to develop a product roadmap and build integrated system for data analytics. Allen Holmes, VP business development at Yellowbrick, said. “Our partnership means that users will enjoy the best of both worlds—the reliability and features of SAS along with the high performance and implementation flexibility that Yellowbrick delivers. This is a win-win collaboration for enterprises.”

Zadara, which supplies on-premises arrays under a managed service scheme, has partnered with colocation operator Cyxtera. The 62 Cyxtera co-lo centres around the globe can have multi-tier Zadara arrays installed in them, available via the Cyxtera Marketplace.

HPE storage revenues take a beating, blames pandemic supply chain woes

By
Chris Mellor
-

HPE’s storage revenues slumped 18 per cent to $1bn in its second 2020 quarter ended April 30 as “component shortages and supply chain disruptions related to the COVID-19 pandemic…impacted our ability to fulfil customer demand”.

Dell reports its Q1 fy2021 earnings, in the quarter ended April 30, on May 28. This will give us a good indication if HPE’s supply chain woes are representative of the computer hardware industry at large. Or if HPE has entered into a world of extra pain.

The company has an order backlog of $1.5bn, exacerbated by component shortages, and twice it previous record, The Register reports. Our sister publication has covered HPE’s disastrous Q2 Fy2020 earnings with slumped revenues, down 16 per cent to $6bn, and a $1bn cost-saving reorganisation in prospect. Here, we explore the storage segment where revenues fell 18 per cent to $1.1bn.

We have charted HPE revenues by segment, below. This is a useful way of illustrating that HPE’s revenue problems for its compute business pre-date the pandemic.

HPE’s storage revenue fall contrasts with IBM’s equivalent quarter which saw 18 per cent storage revenue growth driven by its mainframe refresh cycle. HPE sells XP8 array systems, which attach to mainframes, but did not report any equivalent benefit.

Big Data revenues was a bright spot, growing 61 per cent Y/Y at constant currency – but no actual numbers were revealed. Big Data products used to include StoreOnce dedupe and replication and HPE Data Protector enterprise backup and recovery, but apparently no more.

Storage services bought by Nimble array customers were another positive item – up 20 per cent at constant currency and, again, no actual numbers were disclosed. The composable cloud business – i.e. HPE Synergy – grew one per cent in constant currency.

Meet Arrikto and its Rok container storage backup software

By
Chris Mellor
-

Arrikto, a small California startup, has devised a form of shared storage for Kubernetes-orchestrated workloads. It claims customers can save 64 per cent of AWS costs by running a 100TB Cassandra cluster in AWS using Arrikto’s Rok software, as opposed to Amazon’s Elastic Block Store.

Rok uses local NVMe SSD storage on a cluster node to provide persistent storage to containers. The node is located on-premises or in a public cloud.

Rok snapshots the application’s containers, stores those snapshots in S3-compatible storage and enables sharing across a global infrastructure.

Arrikto snapshots can migrate a containerised workload – apps plus data – across machines in the same cluster or across distinct locations and administrative domains over a decentralised network. This facilitates collaboration.

Rok lives on the side of the local storage stack and acts as a global data distribution layer that makes cloud-native apps permanent and portable. The software-only thin data services layer abstracts the underlying persistent storage and provides instant, parentless clones and snapshots.

Rok distribution layer concept

Customers can take immutable group-consistent snapshots of their applications and keep them in a backup store such as AWS S3. Combined with Rok’s instant cloning, recovery from hardware failure for this setup is quoted in minutes regardless of the stored volume’s capacity.

Rok Registry

Rok Registry is operated via a single pane of glass where customers can search, discover, and share datasets and environments with other users. Users create private or public groups and can define fine-grained access control lists.

Rok Registry brings together many Rok instances in a peer-to-peer network. The management tool does not store the actual data – merely references to data.This allows Rok instances to exchange snapshot pieces over the network. Only the changed parts traverse the network when users create new snapshots.

There is no vendor lock-in and customers can use any type of hardware for compute and storage. They can also move to the cloud provider of their choice, or move among different cloud providers.

Cost reductions

A Cassandra cluster running in AWS typically uses Amazon’s Elastic Block Store (EBS). Let’s envisage a 100TB Cassandra cluster on AWS that uses Rok store, with NVMe SSDs local to each instance. Arrikto claims this is 64 per cent cheaper per month, with a 51x read IOPS advantage. Here is Arrikto’s table:

The same setup on Google Cloud results in a 32 per cent cost advantage for Rok, according to Arrikto.

Citing performance and cost-savings, the company strongly recommends NVMe-backed instances combined with Rok.

Not a virtual SAN

From a data protection point of view, think of Arrikto as being akin to Kasten‘s K10 software, which provides backup and recovery for cloud-native applications and application migration across heterogeneous environments. Arrikto has pretty much the same functionality.

Kasten CEO Niraj Tolia, tells us: “We believe that Arrikto and Kasten occupy separate places on the data management spectrum. Arrikto is focused on primary storage [and] states that it is focused on fast storage for Kubernetes and, like all traditional storage vendors, providing supporting snapshots and cloning is a core part of any storage system’s functionality. 

“… It is critical to have data protection provided by a completely separate system with its own fault domain. This is why Kasten focuses on cloud-native backup, works with a variety of storage vendors, and provides customers the freedom of choice of selecting the best storage vendor for their environment (cloud or on-premises).”

Like Arrikto, Portworx provides storage and data protection for containerised applications. It also offers high-availability, backup, continuous replication and disaster recovery, on top of basic storage provisioning.

Rok does not provide a virtual SAN, such as provided by StorageOS. Apps running in each node in an Arrikto cluster can only access their local storage. They cannot access the SSDs running on another node.

Arrikto background

Arrikto – Greek for ‘explosive’ – was founded in Palo Alto by CEO Constantinos Venetsanopoulos and CTO Vangelis Koukis in 2014. They were engineers in the Hellenic Army and subsequently involved in building the Greek Research and Technology Network’s large-scale public cloud infrastructure.

Funding totals $4m, with a $2m seed round in 2015 and a $2m round in 2019.

Portworx upbeat on container storage revenues

By
Chris Mellor
-

Portworx, the California container storage startup, today issued a so-called momentum release, boasting of customer and revenue growth.

As usual with US startups, the company does not mention actual figures but by any reckoning it is a small fish in the data storage world – annual revenues according to this possibly out-of-date estimate are $14m. However, Portworx’s bullishness is an indicator that the container storage market could shaping up into serious money.

Portworx today said it had “more than 145 customers, including 54 Global 2000 or government accounts”. It reports 136 per cent growth in revenue year over year in Q1 2020, and 92 per cent revenue growth from Q4 2019. Thirteen sales were over $100,000, up from five sales over $100,000 in Q1 2019.

A 2019 Portworx survey showed 87 per cent of respondents said that they used container technologies, compared with 80 per cent in 2018 and 55 per cent in 2017.

Almost Ninety per cent of enterprises are already running container technology and more than half the containers they run are stateful, according to the 451 Research survey: Voice of the Enterprise – DevOps, Security, AI/ML, and Cloud Native – 2020. And those customers need storage.

Background

Portworx emerged from stealth in 2015 and has bagged $55.5m funding over three rounds. Its software runs on commodity servers and aggregates their storage into a virtual SAN providing scale-out block storage. It provides storage from this pool for containers, at container granularity, and with a global namespace. File and object storage are on its roadmap.

Portworx’s pitch is that the storage supplied to containers through an orchestration layer like Kubernetes should be containerised itself and also enterprise class, with features like  security, data protection, backup and recovery, disaster recovery, SLA management, and compliance.

It says traditional enterprise storage, with the features, is suited to virtual server environments but not cloud-native ones, even if they Kubernetes CSI plug-ins. Storage provision for containers has to be supplied at the speed and scale of container instantiation, deployment and removal. Portworx claims that only cloud-native storage, its cloud-native storage, can meet this need – not legacy SANs.

Nutanix snuggles closer to ServiceNow

By
Chris Mellor
-

Nutanix has bolstered support for ServiceNow with the latest iteration of the Calm app automation and management tool and its Prism Pro monitoring SW. The company says the update enables joint customers to reduce IT infrastructure service management costs and simplify processes and procedures.

Rajiv Mirani, Nutanix CTO, said in a statement: “This strengthened integration with ServiceNow, along with the broader suite of Nutanix automation solutions, will allow IT teams to reduce the amount of time they spend on day-to-day management of their cloud infrastructure, as well as applications, so they can focus on supporting business priorities.”

The first Nutanix-ServiceNow integration was announced in October 2019, with ServiceNow automatically discovering Nutanix systems data: HCI clusters, individual hosts, virtual machine (VM) instances, storage pools, configuration parameters and application-centric metrics. ServiceNow users can provision, manage and scale applications via Nutanix Calm blueprints.

With Nutanix Calm 3.0, joint customers use ServiceNow’s approval flow and audit capabilities to automate app lifecycle events such as upgrades, patches, and expansions. Passwords can be maintained in a central CyberArc vault via a CyberArc-ServiceNow integration.

Users of Nutanix Prism Pro management tool are now able to get and respond to alerts and incidents directly in their ServiceNow portal. They can track infrastructure issues through automatically created tickets in ServiceNow and resolve them in this ticketing system.

Joint customers will be able to limit unintended public cloud consumption. They can incorporate Nutanix cost and security optimisation recommendations into existing ServiceNow workflows by automated ticket creation – which is based on cost savings recommendations and security vulnerability alerts – and assign them to the appropriate owner.

Calm 3.0 adds functionality to provision Infrastructure as Code (IaC) with a Python-based domain-specific language (DSL) for writing Calm blueprints. The Calm DSL has the same capabilities as the Calm UI, but is human-readable version-controlled code that can handle most complex application scenarios.

Nutanix says the expanded ServiceNow integrations are available to customers. Calm 3.0 is current under development.

Competition

Nutanix aims to make its hyperconverged system portfolio easier for ServiceNow customers to use than competitor products. The stakes are high as ServiceNow is used by three quarters of the Fortune 500 and dominates Gartner’s 2019 IT Service Management magic quadrant. The message is that, if customers use ServiceNow and want an HCI system, they should buy Nutanix now.

Dell EMC has an OpenManage integration to enable ServiceNow users to monitor and manage their PowerEdge server infrastructure within the ServiceNow console. There appears to be no specific VxRail integration with ServiceNow.

Cisco’s HyperFlex HCI is managed by Cisco’s InterSight, a SaaS infrastructure management tool. There is a ServiceNow ITSM (IT service management) plugin for InterSight, which provides inventory sync and incident management. That means HyperFlex incidents can be tackled through ServiceNow.

HPE’s OneView, the precursor to InfoSight, had an integration with ServiceNow in 2016. It synced hardware catalog entries across platforms, bi-directionally tracked hardware physical events and presented these events as service tickets in the ServiceNow workflow. Blocks & Files understands InfoSight has inherited this capability.

1...345346347...436Page 346 of 436

ABOUT US

Blocks & Files is a storage news, information and analysis site covering storage media, devices from drives through arrays to server-based storage, cloud storage, networking and protocols, data management, suppliers and standards. We publish news, views, opinions and analysis of storage industry events and developments, aiming to keep storage users, developers, consultants, marketeers, salespeople, support staff, analysts and everyone else involved in IT storage up to date with what’s going on.

FOLLOW US

© Situation Publishing, 2018-2024