Criminals need two things to make a ransomware attack work for them; a penetrated IT security system, and inadequate or non-existent backup and disaster recovery.
Sadly, this is an all too common state of affairs, as illustrated by Garmin, the latest high-profile victim of ransomware. At time of writing, the company is slowly restoring its GPS tracker and navigation services. But it is thought to have paid up to $10m to the ransomware attackers to gain access to the decryption key, according to The Times and other reports. The company will have also suffered substantial downtime costs and possibly a hit to its reputation. No-one should blame the victim of crime. But this is a hot mess.
Ransomware is a multi-billion dollar operation in 2020 and IT security vendors are engaged in an arms race with the perpetrators. However, no IT system is 100 per cent secure. This is which is why backup and DR are required as the last line of defence.
In the event of an attack, this line of data protection should not fail. Ransomware victims need readily available backup and disaster recovery files that are uncorrupted by malware. The recovery process should enable victims to quickly rebuild clean systems.
So when it comes to ransomware, backup and recovery is last but not least. Blocks & Files has polled a group of backup and recovery suppliers for their take on ransomware defences. Our contributors are Acronis, Cohesity, Databarracks, Datto, Druva, Rubrik, Unitrends, Veeam, Veritas and Zerto. Commvault, Iland, Pure Storage, Radware and TechData also contributed. We publish their replies below.
How does a supplier protect its customers against ransomware attacks?
Acronis Candid Wüest, VP Cyber Protection Research, said: “Relying on separate backup, vulnerability assessment, patch management, remote management, anti-virus, and anti-ransomware [heuristic] requires an MSP technician to learn and maintain all of those applications. Besides, … Acronis Cyber Protect combines all of these tools into one single product with one single agent, one single management console, and one single license.”
This sounds like an excellent idea.
Pascal Geenens, director of threat intelligence at Radware, tells us: “The most effective way to mitigate the impact of ransomware is offline backup systems, with emphasis on ‘offline’. Any online backup system is at risk of corruption or destruction by the ransomware, keeping backups offline and disconnected from the corporate network will ensure they stay tamper proof.”
He says: “Once the infection vector is identified, it is important to first close the security gap before recovering the systems. Missing this step leaves the door open to re-infection by the same or other ransomware.”
This is a good point to recognise.
Gijsbert Janssen van Doorn, Director of Technical Marketing, Zerto, told us: “The Zerto IT Resilience Platform uses Continuous Data Protection to keep data protected in real time. If – and when – a cyberattack makes it past your defences, your data can be recovered in seconds with just a few clicks.
“Zerto enables organizations to use network, journal, and IOPS statistics to determine the precise moment the ransomware became active and recover to the point within seconds before it happened. Zerto also allows you to quickly perform a failover test to see if you have the right point in time. If not, you can easily failover to a different point, and Zerto’s journal-based recovery lets you recover anything from just a few files to an entire application stack.”
Bob Petrocelli, Datto CTO, tells us: “Local backups are … traditionally the least secure. This is because they are only as strong as the home network, which is inherently vulnerable to attacks and disaster. The same event that may require a backup will likely impact any local protection.” If backups are local then use physically air-gapped ones as part of your ransomware DR strategy.
Ezat Dayeh, SE Manager, Cohesity UK, said: “Cohesity delivers a unique combination of an immutable file system with DataLock capabilities, plus policy-based air-gap and multi-factor authentication to prevent backup data from becoming part of a ransomware attack. Its CCyberScan gives deep visibility into the health and recoverability status of protected snapshots. CyberScan shows each snapshot’s vulnerability index and actionable recommendation to address any software vulnerabilities. This helps you to cleanly and predictably recover from a ransomware attack.
A Pure Storage FlashBade has has immutable snapshots of backup data and associated metadata catalogs created after full backup is performed. They provide a copy of data that a ransomware attacker cannot compromise, alter, or affect.
Rubrik’s Robert Rhame, Director of Market Intelligence, tells us: “Rubrik has an immutable file system and hardened native architecture that attackers can’t flatten. Determining the scope of the damage, point in time, and rebuilding the application dependency chains is automated by Rubrik Polaris Radar. This eliminates guess work during a very stressful event. Rubrik can also live mount for forensic analysis purposes, and then instantly recover workloads to get the organisation back in business.
Michael Cade, senior global technologist at Veeam, said: “There are five functions to help protect customer environments from ransomware, as modelled out by NIST (National Institute of Standards & Technology): identify, protect, detect, respond; and recover. Veeam is able to support each of these areas.”
Veeam stresses the need for immutable and air-gapped backups.
Cade said: ”Veeam has simple, flexible and reliable capabilities built into the recovery process that will securely recover data in the event of ransomware. With as little as checking a box, a service or application will be analysed prior to being brought online for any traces of malicious content. Additionally, there is a default capability to create a “sandbox” of a compromised system to better understand why a compromise occurred and to learn from an event to prevent a future cybersecurity incident.”
There is a downloadable Veeam anti-ransomware cookbook by Product Strategy Director Rick Vanover. It is worth a read.
Doug Matthews, Veritas VP, Product Management said: “A major enemy of successful disaster recovery from large events is complexity. Veritas helps IT teams counter complexity by delivering a single platform that services all their disaster recovery requirements. Our platform delivers recovery technologies to meet every service level objective in the recovery spectrum via orchestrated, tested recovery for physical, virtual or cloud based workloads. In that regard, Veritas is ready to help by protecting data with a single solution for over 800 date sources, writing to over 1,400 storage targets either on-premises or across over 60 cloud providers.”
Yes, this is marketing but it also says Veritas takes an all-in-one approach (see Acronis above) and is ready for ransomware attack recovery action. Its systems are designed to enable fast recovery at enterprise scale, for any granularity (file, instance, application, data centre etc.) with an instant access capability.
Justin Augat, VP Product Marketing at iland, emphasises air-gapping: “All customers get iland Secure Cloud Backup as a Service to enable remote copies of all data (defeating ransomware leverage) and optional “Insider Protection” which is an air-gapped repository accessible only by iland. In the case the customer requires the solution of last resort, iland Secure Disaster Recovery as a Service is a ready, remote environment (protected by security/defence strategies).
Defeated ransomware attacks and lessons
Every supplier we talked to said some of their customers had defeated or recovered from ransomware attacks with their products.
“We cannot name individual customers, but we had many customers that were saved by the Acronis DR solution after a ransomware attack knocked them offline.”
Commvault said ransomware attackers struck the City of Sparks, Nevada. The infection locked police department shared files and left crucial GIS data – think fire and EMS dispatching – unavailable. The Sparks IT team completed data recovery in just 12 hours, not weeks, using Commvault data protection.“ Watch a video about the incident.
An un-named mid-western US City, which was a Datrium DVX customer, suffered a Ryuk ransomware attack in January 2020. An Azure shares was compromised and led to a startup script automatically executing, which began to encrypt all the system files. The entire city was shut down, and the Governor contemplated declaring a state of emergency. Datrium’s support team located a clean DVX snapshot before the attack, which created enlarged snapshots due to its encryption work. This became a golden master and the city’s VMS were restored one by one and verified from this copy. This work took two days and no ransom was paid.
This was an ad hoc recovery, made simpler because a single data centre and a single storage supplier with immutable snapshots were involved.
Rubrik customer the North Carolina City of Durham detected a ransomware attack on Friday, March 6, and recovered over the weekend. They were able to restore critical city services, including access to 911, from the Rubrik backups. Core business systems, including payroll, were back online by the start of the business week. The cleanup involved rebuilding 80 contaminated servers and re-imaging around 1,000 PCs and notebooks.
A second Rubrik customer, Kern Medical Center, discovered a ransomware attack in June 2019. Users reported they couldn’t access their systems. They were able to recover all of the impacted systems protected by Rubrik within minutes, including recovering their business-critical electronic medical record system. Other systems took up to 16 days to be recovered. The customer has now added Rubrik protection to some of its legacy IT systems.
Veeam’s Cade tells us: “A Canadian engineering firm had 100 per cent recovery from a ransomware attack thanks to Veeam and backup data stored in the cloud. Veeam has a dedicated Support SWAT team to handle any cyber security related incidents and assisting in getting customer data back up and running as soon as possible.
Zerto’s Van Doorn said: “TenCate, a multinational textiles company based in the Netherlands, has experienced ransomware twice. The first [CryptoLocker] attack occurred prior to implementing Zerto, and the second occurred after implementing Zerto.
“When the first attack hit, TenCate’s only recovery method was restoring to backup disks, which required a lengthy, multi-step data restore process. Using that method, TenCate experienced 12 hours of data loss and didn’t recover for two weeks.“
“After implementing Zerto, TenCate experienced a second ransomware attack, but this time Zerto’s journaling capabilities allowed the IT team to select a recovery checkpoint a few minutes prior to the ransomware attack. TenCate only experienced 10 seconds of data loss and was able to recover in under 10 minutes.” That’s what we call a decent ransomware attack recovery process.
Veritas’ West said: “We helped a large European energy company who were hit hard during the NotPetya incident. They had a diversified portfolio of solutions protecting their infrastructure at their headquarters but relied on NetBackup appliances as a key element of their data protection strategy.
“When they were attacked with NotPetya, they discovered that, not only was their primary data encrypted, the operating systems of the other backup tools, which were running on a Windows Server architecture, had been encrypted too. The hardened infrastructure of the Veritas appliances meant that this data, however, could be restored efficiently. And, since the customer had chosen to include their Windows Server data in their backup set, the other backup environments could be restored as well. As a result, none of the data in the Head Office estate was lost.”
This, the ransomware encryption of the operating systems of the other backup tools, was a gigantic fail of backup technology. Veritas said its backup appliances are protected (hardened) against malware and have intrusion detection.
According to West, organisations need offsite copies of all valuable data that are complete, current, air gapped and immutable.
Should customers regularly test DR readiness and ability to recover from ransomware attacks?
All the suppliers, without exception, said yes, customers should regularly, and even rigorously, check their ransomware attack recovery plans, with simulated attack recovery events and processes. The aim is to find missing process steps and unprotected devices and applications.
For example, Petrocelli said: ‘Scheduled and frequent testing – and allowing adequate time to resolve any problems that the tests might highlight – is the only way to guarantee that business operations can be restored quickly and smoothly in an emergency.”
The test should simulate a successful ransomware attack with a large and random group of compromised apps and systems; it has to be serious and realistic.
Veritas includes the capability to automate ransomware attack recovery rehearsals in its data protection platform to validate readiness to recover.
Tech Data’s Bart van Moorsel, Security Solutions Architect, EMEA, said: “You must test disaster recovery to make sure everything works. Disaster recovery is your card of last resort. It is a bit like having a life jacket or a fire extinguisher. You hope you never need it, but when you do, it is nearly always critical that it works as intended.”
Ian Schenkel, VP EMEA at FlashPoint, a threat intelligence platform, says: “When it comes to ransomware, the ideal state is to not only be ready to respond in emergency situations but already be prepared so that your team can act quickly. Things like ransomware workshops, tabletop exercises, stimulated scenarios, and engaging with an experienced partner who has dealt with various ransomware incidents are all things that can help prepare an organisation to swiftly mitigate risk associated with ransomware.”
Marc Laliberte, senior security analyst at WatchGuard Technologies, sings off the same hymn sheet: “At a minimum though, you should be testing your backup restoration process quarterly or, preferably monthly, to ensure your systems and procedures are all in working order.”
Does this require different test parameters and methods from a normal DR test?
Zerto’s van Doorn said: “The challenge with ransomware is that it requires both fast and granular recovery, not only from a recovery point perspective, but also from an object perspective, as ransomware can attack only certain components of the infrastructure.”
You should be able to “choose from many recovery points from seconds to days in the past and one that gives you the ability to test the recovery of any type of object (from a single file to an entire datacenter) without impacting the production systems is crucial. Having such a non-intrusive solution in place also allows for easy automation of these recovery tests which results in more frequent tests than just once or twice a year.”
Rubrik’s Rhame stressed a tabletop testing concept: “Most organisations have playbooks for many different Business Continuity Management (BCM) scenarios, but up until a few years ago few had updated them to take all the encrypted non-functioning pieces of a ransomware attack into account. This gap must be closed… Tabletop [DR] testing can expose gaps like lack of out of band communications, uncover insufficient recovery bench strength, and introduce all the organisational stakeholders before they meet during a crisis.”
Augat from iland says an untested ransomware DR plan will likely fail, and asks: “Does your backup solution use AI to automatically detect and alert you to ransomware early on so you can restore quickly? If not, you should look for a solution that can identify if ransomware resides in your backups and select the best backup to use for restores. Lastly, does your backup solution offer testing and verification? Recovery assurance will give you the flexibility to regularly perform testing and verification of your backups.”
Veaam’s Cade reminds us: “Businesses have to be sure that when they are restoring from older backup sets, they have the capability of being able to scan the contents of those backup files with the latest virus definitions and ensure that you are not restoring compromised data into a live environment.”
Veritas’s West says: “Every business should assume that it will be the target of a ransomware attack and plan from there.”
Blocks & Files recommends every business to protect all its data with physically air-gapped, immutable backups. They require analysis facilities to detect clean backups before the attack and rehearsed playbook automation to recover systems.
Organisations with distributed data centres, hybrid cloud operations, and multiple storage and data protection suppliers should realise that they have the most difficult ransomware attack recovery procedure of all. So, they must pay particular and regular attention to rehearsing attacks and attack recovery.
If your executives don’t believe this is necessary, just show them what happened to Garmin.