Ransomware attacks on enterprises are escalating both in frequency and complexity. Many in the security space believe that WannaCry and NotPetya were only a sample of what’s coming. Increasingly, Active Directory (AD) is at the center of cyberattacks, with wipers like MBR-ONI utilizing AD to maximize the attack reach and, in some cases, wipers like NotPetya taking down AD completely. This blog series explores real-life stories of wiper attacks, their impact on AD and what you can do to make your organization more prepared for the next wave of attacks.
Dealing with a Wiper Virus: When NotPetya Attacks
It has been almost a year since the outbreak of the NotPetya wiper attack and, during this time, I’ve had several conversations with system analysts whose organizations were hit by wiper malware. Due to the fact that Semperis is a directory services protection company, most of my conversations are around directory services and the impact of the attacks in the context of AD. In each conversation I’ve had, we talked about what took place and the impact on the organization in a post-mortem view. We then discussed what was done to mitigate the attack impact and the lessons learned, so I thought it might be beneficial to others to hear some of these stories to be more prepared for future attacks.
The first story I want to share about NotPetya is of a large manufacturer in Western Europe, with roughly 10K employees, whose entire Active Directory was encrypted as a result of NotPetya. What is astonishing about this story is that, from the moment the wiper started spreading, it took it less than 10 minutes to encrypt all of the company’s Domain Controllers (DC) and file servers (FS), with the encryption spreading globally to all of the remote sites.
Unfortunately, the backup and recovery servers were also encrypted and had to be rebuilt from scratch. The attack brought the organization to a complete standstill and, after the IT teams reviewed the initial damage assessment, they realized that this attack was like nothing they had ever seen before.
The restoration process that followed the attack was long and complex. First the team needed to make a few critical decisions:
- What should be restored first?
This included mapping out the business-critical applications and the required infrastructure components that they rely upon.
- For each site, what is the minimum number of DCs that needs to be restored?
In addition, special attention had to be given to restoration of servers that were in low-bandwidth locations.
The first step in the recovery process was rebuilding the backup and recovery servers, followed by the restoration of the other servers from tapes, a process which took several days with Microsoft engineers on site for support.
Restoring the encrypted DCs was a tedious process that needed to be executed from scratch. The IT team started by spinning a new virtual machine with a clean operating system, installing the recovery agent, performing a system state restore, cleaning the virtual Network Interface Controller (NIC) and resolving other driver-related issues following the system state restore.
Even though the team followed the proper restoration protocols, the initial restoration of Active Directory failed due to an update sequence number (USN) rollback issue that prevented AD from replicating. The challenge with USN rollbacks is that they are very hard to identify and there can be thousands of them, with no error logged anywhere.
Expecting the Unexpected in the Recovery Process
The key takeaway from this story is that recovering an encrypted environment is an incredibly complex process, with many moving parts to be considered.
Even when you think you’re truly prepared, it’s hard to predict some of the issues you could run into during the restoration. Here are just a few ways you can better protect your organization from the effects of a wiper virus:
- Always prepare for the worst-case scenario – Even though it seems unlikely, you should be prepared for a situation where all your servers, including the backup/recovery servers, are gone. Put into place a procedure where the recovery of the backup/recovery servers is the first step.
- Map mission critical applications and their infrastructure dependencies – Most likely your line of business applications rely on Active Directory, but it’s important to know what other components have to be restored for the application to run.
- Make sure that you have a tried-and-true plan to restore your Active Directory – Having a backup is a great first step, but you need to be aware that recovery is not a straightforward process and things can get complicated very quickly. Microsoft offers an Active Directory recovery preparation service called ADRES, that can help you to be prepared for a disastrous situation with your AD.
Taking the Pain out of NotPetya Recovery
While it’s critical to take certain steps to protect your infrastructure against wiper attacks, having a solid Disaster Recovery plan in place is the only way to truly minimize the complexity of the recovery process. Semperis’ Active Directory Forest Recovery (ADFR) solution fully automates the orchestration of the Active Directory Disaster Recovery process and dramatically reduces the time to restore. By leveraging internal intelligence, ADFR is able to make decisions that solve for a variety of issues that can arise during the AD DR recovery process (for example, using distribution points to overcome the low network bandwidth problem). For additional information on Active Directory Disaster Recovery, you can view our webinar on AD DR automation and our webinar on AD backup considerations.
Stay tuned for the next blog in the series, where I will talk a little bit more about the ONI virus and its effect on Active Directory.