Wenn jede Minute zählt Die Wiederherstellung von AD während eines Angriffs

Hi there. I’m Tom Field. I’m Senior Vice President of Editorial with Information Security Media Group. Very pleased to welcome you to our topic today. The session is When Every Minute Counts, Recovering Active Directory During An Attack. Leading our discussion today is Sean Deuby. He’s Principal Technologist with Semperis. Now before I bring Sean onto our virtual stage here, a little bit of background. What do you do when your endpoint or other initial protections are breached? When intruders are already in your network and have already compromised your Active Directory forest, this is the true story of a ransomware attack and how it was stopped. Sean is gonna share lessons from this real live incident response and attack intervention. You will learn why you should prioritize AD protection and recovery, how and why ransomware attacks target AD, and how to recover AD securely during the attack in progress. A little bit of background on my organization. Information Security Media Group is a global education and intelligence firm. We’re based in the US in Princeton, New Jersey, and, of course, you may know us by any number of our thirty seven media properties. These include GovInfo Security, Data Breach Today, and BankInfo Security. In all, we serve an audience of over one point three million security and technology leaders globally, and we give them a daily diet of news, analysis, research, events, and educational programs just like this one. Few notes of housekeeping. If you have any questions for Sean, you can submit them anytime by the chat window on your screen. I might not be able to get to every question today, but I promise you this. Any question we can’t get answered in the course of this session, we will get you a response afterwards via email. Should you encounter technical issues while trying to view today’s webinar, take down that email address you see on your screen. If you write to webinars at i s m g dot i o, we’ve got support staff standing by to help. And a reminder, today’s webinar is copyrighted material meant for today’s session and individual study purposes only. If you’d like to use any of the information presented today or if you’re looking for customized training materials, please contact us. Delighted to introduce our sponsor, Semperis, For security teams charged with defending hybrid and multi cloud environments, Semperis ensures the integrity and availability of critical enterprise directory services at every step in the cyber kill chain and cuts recovery time by ninety percent. Purpose built for securing hybrid identity environments, including Active Directory, Entra ID, and Okta, Semperis patented technology protects more than one hundred million identities from cyberattacks, data breaches, and operational errors. Now let’s meet Sean Deuby. He brings more than thirty years experience in Enterprise IT and hybrid identity security to his role as Semperis Principal Technologist North America, an original architect and technical leader of Intel’s Active Directory, Texas Instruments Windows NT network, and fifteen time MVP alumnus. Sean’s been involved with Microsoft Identity Technology since its inception. And with that, let’s bring Sean Deuby on board for our conversation here. Sean, thank you so much for being part of this today. Yeah. Thanks for having me. Thanks for, getting together today and, chatting about this topic, which is challenging. It’s a challenging thing to be sure. Now, Sean, I know you’re gonna take us into a specific, attack here and and how you walk through it. Let me ask you this. When we talk about attacks involving Active Directory, what are the most common types of strikes you see? I would think ransomware would be right at the top of the list. Oh, well, certainly ransomware as a tactic for sure. And what I love to say is that, and time has proven me out. If you wonder what the threat actors are going to do, just look at what their motives are. And in the case of ransomware actors, it’s to make as much money as possible as quickly as possible with as little interference from law enforcement as possible. So we started out with single extortion or just extortion where they would encrypt them encrypt the environment. And then the most popular right now is double extortion where, you know, data is exfiltrated and then they encrypt. But they’re continually inventing ways to force compliance. And what we’re starting to see now in some areas is actual physical coercion as well, which is pretty scary. So yeah. Sean, are you gonna share a timeline with two different views, the production network and the isolated network? What is it you’re going to attempt to show us? Well, certainly, one of the things that is the goal of this time together is to show you how complicated it actually is to recover Active Directory during a cyber attack. There are a lot of things to think about, and, I think of the old phrase, don’t try this at home. And maybe this is actually the thing that you should try at home. But, even at the level that I’m describing, I’m leaving out a whole lot of detail. And, I’ll sort of cover some of those details at the end, but, this shows you what it’s all about. This shows you gives you an idea of the complexity of what it is to defend and recover, an Active Directory forest during a cyberattack. Well, let’s get going, shall we? I want you to set the stage for your timeline of an attack and begin with what does the incident response team have to do first? Sure. So let me, as it was you suggested, let me share the screen out. So to set the stage, what had happened is, well, we were contacted by one of our partners that, there was an offshore organization, and I’ll just leave it at that. Pretty large organization that was, suffering from a cyberattack and, a fairly significant one as well. The partner that we are working with, had been working with them for quite a while, and, it became apparent that they were in a world of hurt. And their environment they eventually the partner eventually discovered that there were four different threat actors in the environment. So that’s a bad day when there’s four different threat actors in the environment. And part of the reason there were four different actors in there is because the Active Directory environment was quite insecure. It would had been in production for a number of years, and people over time had taken a lot of shortcuts to make things easier and basically making it very, very vulnerable. And that’s some of the things that I’ve seen from talking to other defenders that this is a very common this is a very common circumstance. As a matter of fact, I had a conversation this morning with someone from our incident response team, and they said in their experiences, oftentimes, it was because the security on their Active Directory was very, very bad. Not just, you know, not Sterling, but bad. So the goal, of they called us in is to restore the trustworthiness of Active Directory, and that involves several different parts. Now this is a big and complicated screen, and it’s crazy. And I don’t need you to try to understand it better or make sense of it because I’m going to go through it as you suggested, step by step. And to your point, to the top of the slide, you can see there is, what’s called the comp we have the compromised network. And in the bottom of the slide is an isolated network. In splitting the two up is a timeline of about how many days it took to do this. So it took a little bit more than six days, and that was six days with some of our top four hundred level Active Directory experts working on it and sort of a follow the sun support mode. So, here’s where it starts. Here is your compromised Active Directory forest. So there are several aspects to getting this working again, and the difficulties in are different than there would be for a typical for a typical file server or or something like that because of all the changes that go on in Active Directory. So when you have an infected forest, the fallout happens in what I’d call the obvious and the not so obvious. So the obvious effects are if you’re in the environment and you get a call from, you know, your worldwide command center or SOC, you’ll say some of our domain controllers have stopped working. Things are just not working like they’re supposed to. Users aren’t being authenticated, whatever. And, so they may be encrypted and maybe malware otherwise messing with the environment. That’s generally the first thing that you understand that something’s happened. And, almost certainly, these domain controllers have malware sitting on them, either in the operating system or in a section of Active Directory that’s called SysVol. So that’s the first thing, and that’s the obvious thing that you see. This the underlying issues that are not so obvious is that the Active Directory service itself has been tampered with. So users threat actors may have added themselves into privileged, administrative groups. They may have changed permissions, and the Active Directory permission structure is immensely complex. So it’s easy to make small changes that will pass notice, by it unless you run really, really thorough scans. This particular example I have is a is an object called admin SD holder, and that controls what permissions are set on privileged objects. So a threat actor will often go after that to loosen up permissions on privileged groups and users. And the other is group policy objects, may have been changed. So group policy is a configuration control in Active Directory, and I like to say it can be used for good to configure everything to look the same, or it can be used for evil to spread software, across the network because it has that capability. So that’s another aspect that the bad guys may have changed in this compromised forest. So with this forest, the first thing that we do is we take a backup of it, as soon as possible, and what we call them, a minimum configuration. This means, you know, the organization may already have Active Directory installed, ADFR installed. That’s our, backup and recovery utility. If they don’t, we just go in and we install it, and we do what’s called a minimum viable, minimum MVP, minimum viable product. And we get agents on one or two domain controllers in every domain if we can, and then, have a stand up on ADFR management server and then back it up right away, before the bad guys can destroy or do anything else like that. So we know at this point, we’ve got a copy of their Active Directory that’s safe should they encrypt the environment. And, Sean, I might be getting ahead of you here. Let me know if I am. But at this point, are you looking to restore the forest on the fresh servers and the isolated network? Funny, you should mention that. Yes. So with ADFR, so the process goes and here, this is the picture version of it. The process is that you wanna create an isolated network away from the threat actors in the environment and restore Active Directory into that isolated network. Now with ADFR, we back up Active Directory and not the operating system. So the process that you would follow is to create, an isolated network, like I said, and provision fresh virtual machines in there. They could be physical machines, but they’re newly provisioned, so you know that there’s no malware, on the virtual machines. And then you install agents, ADFR agents, on the virtual machines. And then you take the management server that has made that backup, and you make it available in the isolated network. Now you could pick it up and move it. You could if it’s a virtual, you could clone it into the individual into the isolated network. There are a number of different ways you can do that. So now what you have are fresh virtual machines, the ADFR management server in there on the isolated network. And then with ADFR, you execute a forest recovery, which is literally a six click process to restore the forest onto those virtual machines in the isolated network. So what you have is a you have a good forest here, but what you don’t have, you still don’t trust the service necessarily because the Active Directory objects are still contaminated. So you’re partly there, but you’re not all the way there. So this is what that object of that process of what we were what we call the forest recovery process. Yep. And executing this manually is a is not a process that anyone I have ever talked to wants to go through more than once. And what Active Directory Forest Recovery automates is this twenty eight step process from Microsoft that describes essentially how to take an Active Directory forest, break it down into its smallest components, reset a bunch of aspects of it, and then start building it back out with more and more servers. And, besides taking a long time, being very complicated, you can see in these steps and many of these steps have to be have to be executed on individual servers as well. Conventional backups don’t help you. They help you on one step, which is to do this individual recovery of the first domain controller in every domain. Other than that, you’ve gotta do it manually. So at this point, we have got, the backup in place. The domain controllers are clean. We don’t necessarily trust the objects, the Active Directory service. So we have to do an analysis. At this point, what do we wanna do? Do we want to keep the existing forest and try to fix it, or do we want to take that forest that we have installed in the isolated environment, clean it up, and use that one instead? It’s a big decision, because it’s a big decision. What are the what are the key factors here? What are you weighing? You know, a lot of it has to do with how badly screwed up the production forest is, number one. Number two is threat actor visibility. There were four different threat actors in this environment. If you start fighting back against the threat actors and they get the upper hand, they can make the environment worse than it is. And a great example is one year ago in November, MGM Resorts were attacked by the the group called Scattered Spider, and they tried to fight back even though Scattered Spider owned the environment. And Scattered Spider proceeded to go in and, using their Active Directory rights, go into their VMware ESXi environment and encrypt all of their ESXi hosts. Thousands of virtual machines. So you have to weigh that. The advantage of doing this work offline in the isolated network is that the threat actors don’t know that you’re doing the work. So in this case, the decision that was made was to leave the compromised Active Directory environment alone and harden the isolated network that we’ll call the recovery forest in preparation for bringing that into production. So a little bit of detail. What does that look like if you harden the recovery forest? So it’s tools, it’s best practices. It’s looking at the most egregious, changes that were made and that make the environment vulnerable. For example, one right off the top of my head that’s a common one is call is Kerberoasting. So more and more people are hearing about Kerberoasting now, and Kerberoasting is a reasonably straightforward process that takes your service accounts in your environment, and every Active Directory environment has service accounts. And every most Active Directory environments have service accounts with privileges. And Kerberoasting is a technique by which you if the service account has a weak password, which many of them do because once you set a service account’s password, you’re afraid to change it. Because when you change it, it breaks the services that depend on it. So, historically, service accounts tend to be old. They tend to have very weak passwords, and they haven’t been changed for a long time. So Kerberoasting allows you to very quickly crack the hash of that password of that service account, and then immediately give you the privileges that that service account had. So there is a report, I think it was from CrowdStrike, a few months ago, and they said that they detected a five hundred and eighty three percent increase in Kerberoasting for that exact that exact reason, for the way you can break into it. So we’ve got this forest. The infrastructure is trustworthy, but the service is not trustworthy. So you have to clean it. So before you can clean it, you have to figure out what are the bad guys doing there. So we have a tool that comes along with Active Directory Forest Recovery called Purple Knight Post Breach Edition. That’s separate from the community Purple Knight, and it focuses on looking for indicators of attack. You turn you turn loose Purple Knight in this recovery forest, and then you set the time frame for about when you think the attack happened and and look for the changes in the environment. So as I said, the beauty of having it in this isolated environment is that you can, I say, take your time? I mean, every time is a fact is super important. But you can afford to do good forensics without worrying about the threat actor seeing what you’re doing. So you can determine all of the changes that the threat actor has made, and then you can undo those changes to clean up the environment, remove the accounts that they’ve had in, look at their look if they’ve, established persistence, in various nefarious ways in the environment and eject the changes. And you can also run the community edition of Purple Knight, which has got a hundred and fifty different indicators of exposure in it, which can show you four configurations, that you can sometimes very easily take care of. Now in this situation, this is where, of course and you’re doing your forensics, you wanna try to figure out as best as possible what the threat actor has done to get into the environment and specifically to get into Active Directory. Obviously, we’re restricting the scope of this discussion to Active Directory. When a threat actor goes into the environment, there’s a lot more involved in Active Directory. And I don’t pretend to say that this ejects the threat actor out of your environment. We’re just talking about recovering Active Directory. So remediate the most critical items, both from Purple Knight, Post Breach and from regular post breach. And then if you have time, apply security best practices as well, such as administrative tiering, where the tier zero assets that control Active Directory, lower tiers like servers and workstations don’t have access to it. That doesn’t always happen. What we have found is it’s like, get this thing back up, and we’ll worry about improving the, we’ll worry about improving some of the administrative practices once we’re back up. I should mention that, literally ten days ago, maybe ten days ago, the Five Eyes intelligence community. So the Five Eyes, if you’re not familiar with it, is the intelligence services of the United States, Canada, United Kingdom, Australia, and New Zealand issued a joint advisory, an urgent advisory on specifically on Active Directory called Detecting and Mitigating Active Directory Compromises. It’s like a seventy page document telling you basically what we’ve been trying to say for a long time is how important it is and how critical it is and how often it’s attacked. And it gives a list of seventeen of the different most common attacks against Active Directory. And of the three tools that are mentioned in there to help you in that document, one of them is Purple Knight. So the Five Eyes say, go download Purple Knight and look for it. So, Sean, again, I might be getting ahead of you here, but what I’m gathering is, okay, you work in isolation. At what point do you bring your restoration into production and how? This is where it gets exciting from a operational viewpoint. Because, obviously, if you look at the overall situation, the environment is in some kind of turmoil right now because of the back and forth that’s going on. So from a technical viewpoint, if you think about the member servers in the environment in the when I say member servers, they’re they’re members of the Active Directory forest. And the clients, the PCs, Macs that use Active Directory, they’re all joined to Active Directory, and they’re all depending on those that compromised Active Directory. What they do is they, they learn how to access what domain controllers they can access through DNS, the domain name service. And Active Directory domain controllers publish specific records in DNS that advertise their services to anyone that wants to look for them. In other words, clients of the domain. So that’s the key. That’s the key of what you wanna be able to look at. So we’ve hardened this directory. We’ve hardened this recovery forest at this point. We’ve cleaned out the privileged groups. We’ve maybe put in a tiering model, all your permissions, and all that. It’s good. It feels like it’s good, and it’s ready to go. How is you say, do we cut it over? So here’s where everybody crosses their fingers, and you have to, essentially, go out and shut down your production Active Directory forest. And, of course, at that point, nobody’s working. That’s a pretty pretty big deal. The next thing that you have to do, and I actually have this in detail. I have the steps in here, detailed out. So we shut down the existing production forest. Then in the recovery forest, you update the records, the DNS resolver records, for the domain controllers to tell them where they should register their stuff. So you may have Active Directory integrated DNS, in which case it’s a little bit simpler. But if you have an external DNS provider, like so many organizations have moved to, BlueCat or some other provider, And they have it has its whole infrastructure with its IP addresses. So you put those DNS server IP addresses in the domain controllers saying, okay. The next time you need to register, you’re gonna register yourself over there. The next thing that happens is you open up your isolated network to the production network. You never you do not want to have two Active Directory forests visible to the network at the same time. Bad things happen crossing the streams as it were. So then, because you’ve got no production Active Directory forest at this point, you tell the recovery forest domain controllers to register themselves in the production DNS. And most of the way that’s done is the technical aspect of it is you go in and you restart the net log on service on the domain controllers. And what that will do is it will force them to find their DNS and then register what are called SRV records and other records in DNS. So at this point, the production DNS server has the right records for this new recovery forest. But the clients and the member servers don’t see it yet. If there is a refresh period and if you sat there long enough, they would come back, but you don’t have time at this point. You don’t wanna wait. So what you do is you reboot everything. So when you go through and you again, this is not your typical operation for the world right now. You go through and you re reboot the member servers and the PCs. They query DNS as part of the reboot. They say, give me the SRV records for the domain controllers that I need. DNS serves them the new ones, and they start using the new forest. So at a high level, that’s the process that took place. Now as you can guess, I left out a lot of things. Well, that was my question, actually. What is missing or not fully explained in your timeline? Oh, gosh. Okay. So a lot. So the first are networking complexities. You’ve got this isolated network. Let’s just assume you have a global corporation or a geographically widespread organization. You’ve got this isolated network, and you say, expose it to the production, to the production network. Open up the firewall. Well, how does that work? I mean, how do you have traffic that is able to route from that just opened up network across your whole corporate network? And how do you do it efficiently? Another aspect of this is capacity. So when you restore a a forest for troubleshooting, you typically again, it depends on the size of your organization. You have five fifteen five ten fifteen twenty, thirty domain controllers around the world servicing Active Directory. And what you have, once you’ve recovered Active Directory, that isolated network is maybe three or four. You probably don’t have twenty in there. So you run into a capacity issue. Because if you have all the clients reboot all at the same time, you’re going to have a user and server community that is used to twenty five domain controllers hitting five of them, and then, you know, bad things happen. Now, ADFR version four point two, which is coming out, shortly, has a capability called staged recovery, where you can do an initial recovery in this small seed forest like I described. But ADFR has knowledge of how many domain controllers are really out there. So anytime up to, what, maybe ninety days afterwards, you can execute a second stage recovery and do what’s called a DC promo to promote other domain controllers up to help build out capacity. So you can see there’s all sorts of timing aspects for this, that are clearly not comprehended in a forty five minute or hour long presentation. Sean, what would you say the biggest lessons learned are from this exercise? What sticks out to you? Well, clearly, that it is complicated, and that most organizations, when they’re thinking about cyber recovery, don’t think about the subtleties and the unusual requirements for Active Directory to to bring Active Directory up quickly. Many organizations either don’t know anything about forest recovery or they greatly undersell the complexity and the difficulty of doing a forest recovery when everybody’s under stress. I would say another aspect of this complication is depends on how messed up your AD environment is. So for the example of ADFR, we can only recover the domains that we back up. So if you have a forest that has four domains and all of the domain controllers in one of those domains is compromised, you’re not gonna get one of those domains back. It’s it’s just as simple as that. I’ve already said this just talks about the scope of AD recovery. We’re not talking about any other aspects of incident response here. Another one, to be aware of is and they are always out there. You know, as an Active Directory administrator, it’s really difficult to tell what applications and what users rely on your service. It’s very opaque. But applications that relied on the IP addresses of domain controllers because they’ve been stable for so long instead of DNS will all break when they change. And you won’t know what those are until you bring up your new forest and those applications don’t come back up because they weren’t working. So sure. Yes. Go ahead. No. Continue. I have a question for when you get done here. Sure. Cutover strategies are, have to do with and this is a relatively minor one of how you move your management server from production over to the isolated network. Not hard to do. There are different ways to do it, and we can help you with that. But this is a big one. Resynchronizing with your cloud identity provider, and let’s say it’s Entra ID. When you’ve recovered Active Directory back in time or let me back up myself and say, if you think of Active Directory now or what we like to say is hybrid Active Directory as this holistic organism that consists of an on premises component, a synchronization component, and a cloud component. They’re tied together by this cloud synchronization and by certain anchor objects and identities that say this on premises is equal to this in the cloud, and it has these permissions. When you recover Active Directory back to a previous point in time, you’re out of synchronization. And really bad things can happen if you just don’t keep that in mind and you go, oh, that’s right. I need to fix my Entra ID. It’s actually quite complicated. So that is actually a topic of about an hour, an hour and a half session that one of our subject matter experts is gonna give at our our HIP Conference in New Orleans on how do you reconnect Azure AD and Active Directory together when you’ve done a recovery. And the the last thing I’ll say, Tom, is to examine all of your dependencies. You’d be astonished. Most organizations are astonished when they discover the things that are dependent on Active Directory, like their disaster recovery plans. Their disaster recovery plans are stored on a file server that is domain joined to Active Directory and requires their Active Directory permissions to get in to get to the disaster recovery plans or physical security to get into the machine room at the at the data center or any number of things. You’re a privileged identity manager. Yeah. You print the thing out and print them out several places so you have access to it. So, Sean, I wanna ask you about Semperis. How are you helping your customers to proactively prepare to recover from such attacks as you’ve described to us here today? Well, certainly, one of the first ways is with Purple Knight. As I say, a free tool. It doesn’t require any rights in your forest or your domain, and it doesn’t even require any rights on your client. We don’t see any of the aspects, and we don’t see anything about it, but it will do a security analysis of your Active Directory and Entra ID environment or Okta for that matter of about a hundred and fifty different security indicators. And it will spit out literally a seventy five page report with detailed results ordered by criticality, telling you where they fit in the MITRE ATTACK framework and giving you the exact instances where the issue was occurring is occurring and steps that you can take to remediate it. That’s one aspect of it. Another aspect of it, especially in this situation, is, our team, for breach preparedness and incident response. So at Semperis, I think we’re now up to a hundred and seventy cumulative years of Active Directory MVPs and twenty five or thirty former Microsoft, premier field engineers, premier alliance field engineers in Active Directory. So we have a massive amount of Active Directory talent. One of the capabilities that we, offer is a security assessment. We have both a light assessment and a very, very thorough security assessment down to your security processes, how they apply to Active Directory, what your Active Directory governance is, and your disaster recovery planning for consult that we can consult on and help you out with. The other, of course, as I have described, is incident response. Be able to call in and help us out help you out in incident response and in post incident forensics. As we say, and it’s up there on our website. We wanna be a force for good to help organizations, prepare for it, to help them respond if they’re in that situation, and we will be on the phone with you. You have our team support. It’s not just, oh, here’s the product. Here’s the installation manual. Knock yourself out. Not at all. We are there with you if you find that you should need to use these tools in anger. So another question for you before we leave our audience here. Well, let me give you a chance to summarize here. I’ve got a question for you when you get done. Sure. I think I hopefully conveyed this pretty clearly. Responding to a cyberattack with Active Directory and making it safe again is extremely complicated, and it is not something that you want to encounter for the first time when you find yourself when you get that phone call at two o’clock in the morning. It’s restoring both the servers that eighty sits on, the domain controllers, and getting the service trustworthy again. The forest recovery process is extraordinarily complicated just from looking at the documentation, and the documentation is on the web right now. But I went through the exercise of printing well, not printing it out, but making them into PDFs. All of that reference into one big PDF, and it came out to be a hundred and fifty pages long. Again, not something you wanna encounter, at two o’clock on a Sunday morning. And our solutions, ADFR for Forest Recovery, Purple Knight, and the Post Breach Edition, and a tool that I didn’t talk about, Forest Druid, which is an attack path analysis tool that looks from what your most privileged objects are in Active Directory, your tier zero objects, and looks for paths to your tier zero, ways that might be that you might be attacked in ways that you might not have previously suspected. So, overall, to reduce your time to recovery if it happens because as I said, every minute counts when you’re under attack. Sean, we’ve shared a lot of information in a short period of time. Let me ask this last question. What are the questions that our audience members need to be asking about their own capabilities now to recover AD during an attack? Does number one, do you have an a disaster recovery plan for Active Directory? Many organizations still don’t. Do you have a cyber attack focused recovery plan for Active Directory? Because traditional disaster recovery plans don’t suffice for situations like this. Have you looked at the Microsoft Forest Recovery documentation? Have you taken that generic documentation and made it unique to your environment? Go here. Hit the server. Do this. Do this. Do this. And then have you tested it? And if you’ve tested it, when’s the last time you’ve tested it? And you get to the end of that whole long exercise, then maybe it’s worth looking into automating that process, in case you need to use it at frankly, the most stressful time of your IT career is doing incident response. Well, Sean, you’ve offered some invaluable advice, and you’ve illustrated it nicely. Thank you so much for your time, for your insight, and for walking us through this exercise. Thank you, Tom. I appreciate the time. First, I wanna thank our attendees as well. You took time out of your day to attend this session. We’re grateful for that, and I trust you’re walking away with some excellent new strategies you can immediately put to work. As always, I look forward to seeing you again at one of our upcoming sessions. And until then, for Information Security Media Group, I’m Tom Field. Thank you for your time and attention today.