Participating in a simulated ransomware attack showed up the glaring need for attack response playbooks and insulated clean room recovery facilities, as even immutable backups could contain undetected ransomware-infected data.
Cohesity runs Ransomware Resilience Workshops in which participants take on exec roles in a business that experiences a cyberattack that encrypts its data. The business had no pre-existing ransomware attack response plan and the workshop experience was eye-opening.
The workshop was run by Mark Molyneux, Cohesity’s EMEA CTO and an ex-director for global storage, virtualization, and cloud at Barclays Bank. He had set up a scenario in which a fictional retail business, which they dubbed “Dan’s Doughnuts”, with 12,900 stores worldwide and a $1.3 billion annual turnover, was hit by ransomware. Attackers encrypted files, exposed staff identity and system monitoring facilities, and then sent a message to company execs, saying they had 48 hours to pay a ransom of more than $1 million in Bitcoin:
The IT admin team couldn’t initially access the datacenter building, as the computer-controlled entry system was not functioning, and nor could they monitor datacenter operations remotely as that application was down. Most business IT operations, including finance and product distribution and billing, were down. Management was effectively blindfolded.
We attendees took on roles such as the business’s CEO, CIO, CISO, chief legal officer, and PR head with prepared background motivations and responses to each other’s questions. As we grappled with what to do, an attacker called Igor sent a series of threatening messages upping the tension and stakes.
What was immediately and devastatingly apparent was that we, as a mock exec team, had no idea how to cope. Non-IT people had to understand the role of a NOC SharePoint server, domain controllers, and so forth, and IT people were confronted with business implications, such as lost revenue, a damaged reputation, and legal risks in paying Bitcoin – how does a business even buy Bitcoin? – to an entity that may be based in a sanctioned country.
Eventually, the company’s board became involved and were aggressively angry with what they perceived as management shortcomings affecting stock price.
It became abundantly clear that asking business execs and managers to organize an effective response on the fly was chaotic and stressful. Identifying the affected business systems and recovery of data, even with a ransom paid to get encryption keys, was lengthy and time-consuming. Nothing could be restored directly into production environments as you cannot take the risk that the restored data, even if immutable, had been compromised as well.
There wasn’t even a plan to specify in what order IT systems and apps had to be recovered. Of course not; it was a simulation exercise. But does your organization actually have a ransomware attack playbook that specifies these things?
Who is responsible for what? And how they can be contacted out of hours? Who are the substitutes if the main people are unavailable? And do they know they must never switch off their work phones?
Data recovery was slowed because immutable backups may contain unclean data infected by ransomware that is newer than the ransomware detection engine used to scan backups for the last known good copy. The immutable copy has to be restored to a clean room and checked before it is released to production use. There is simply no silver restore bullet from immutable backups.
Tape backup restores take a long, long time, and they too have to pass through a clean room cycle.
The first critical lesson we learnt is that you have to have a plan, and the best way to build that is to experience a simulated attack. It has to be as close to reality as can be, and then assess what needs to be done, in what sequence and by whom, in order to get a fallback system in place and understand the minimum viable set of critical business systems required to operate.
This is not a trivial exercise and it had best involve board members so they too can understand the pressures and needs inside the business and how best they can help.
A second vital lesson is employee phishing attack training. Such attacks should be detected and reported. The training should encompass employees in partner organizations who can access your systems – and it should be ongoing.
How did we attendees do in our simulated attack? I’m not going to say as the result was rather embarrassing.
Bootnotes
- Cohesity offers a ransomware risk assessment facility.
- Several organizations offer ransomware attack simulation facilities, including Akamai (Infection Monkey), Core Security (Core Impact), CovertSwarm, CyberSecOp, Knight Labs, KnowBe4 (RanSim), ManageEngine (Endpoint Central), NetSPI, SISA (ProACT MDR Solution), and Strobes Security.
- A US government CISA ransomware attack checklist can be found here.
- A UK government ransomware attack document which describes effects on attacked organizations and individuals can be read here.
- A video about a ransomware attack by Chief Security Scientist Joseph Carlson can be viewed here.
