Recovering Active Directory: The Missing Piece in Your Operational Resilience Plan

Good morning, good afternoon, or good evening depending on where in the world you are as you are joining us for this event. My name is John Miri. I’m president of the Electric Grid Cyber Security Alliance, and I am very excited to serve as your moderator and host for today’s webcast. Thank you for joining us. We’re gonna have a great session over the next, sixty minutes together. Before we start, I’m gonna read a few housekeeping notes for you about how to make the most of this event. We will email a recording of this presentation to every one of you who registered within forty eight hours. So please use the recording for your reference. You can also share it with any of your colleagues if you want. There’s a share widget in the console menu. We’ve also reserved some time at the end to take your questions. If a question strikes you at any point in the presentation, you can go ahead and enter it in the, Q and A box. We’ll take those in, and then we’ll batch those up and read them all at the, end of the, of the webinar. So I’m joined by two fantastic speakers. Sean Deuby is the Principal Technologist for North America for Semperis, and then Alix Weaver is a Senior Solutions Architect at Semperis. Fantastic folks who have been in this space, doing Active Directory and identity management for a phenomenal number of years since, really since since the beginning. So before we get started, I would like to take a little bit of a poll to hear, a little bit about some of your, your experiences and interests. And the poll question is this, do you have a plan to move, to migrate away from Active Directory, for example, to a fully cloud based infrastructure? And if so, when? So one year, three years, five years, or never. So couple options there. I’m gonna give you a couple, you know, seconds here to vote on that. And before we get started, I’ll frame the discussion a little bit. So if we look at the pages of Government Technology magazine in the past just seven days, it is full of articles about city, county, local governments having cyberattacks. One county had four point six million records compromised, including a voter registration, list just a few weeks before one of the nation’s most pivotal elections. We saw a city have its police records compromised. Hackers demanded a six million dollar ransom from a city airport. As of now, they haven’t paid, and as of now, they still don’t have their data back either. So, that’s just a few headlines in a single week, and we’re all really trying hard to stay out of those headlines even though cyberattacks are everywhere. Now you probably know what a cyberattack would mean for you. When we think about all of the different entities that are on this call today, it could mean a lot of different things. Maybe it means if you’re an electric utility that the power goes out. Maybe it’s on the hottest day of the year or the coldest. That alone could impact the lives of the people that you support. Maybe emergency services or nine one one dispatch could be taken down. Twenty three cities in Texas experienced that, little over a year ago. Maybe it’s the tax office, right, that gets hit where its virtual doors get closed, you’re not able to collect taxes, or the treasury itself gets raided. And then think about transportation. Right? Shutting down traffic signals and bus systems. Kinda sounds like if you told me this stuff ten years ago, I would have thought you’re telling me about some Tom Clancy thriller. But this is a reality that we’re all facing today, and I appreciate you jumping on this call to help learn and get better at this. So today’s call is really gonna be a lot of good news about this problem. There are solutions out there that can help you deal with these challenges, and we’re gonna talk about a number of them today. So on today’s webinar, we’re gonna do a deep dive into how an identity first security strategy and a rapid AD recovery plan plan can safeguard your organization’s resilience, especially against the escalating threat of ransomware. And I think this area has been one of the most overlooked and yet critical aspects of any organization’s resilience, recovering your your identity system, specifically Microsoft Active Directory. If you can’t log in, you can’t do anything else, and so that’s a pretty critical, step in the recovery chain for our organizations. Ransomware in particular and cyberattacks really in general have become these top tier threats. When we invest in all of these things like endpoint security, cybersecurity insurance, even paying the ransoms, all of those things can be important. But for most organizations, AD is really the backbone of where it all starts. And for many of us, we don’t have a strategy for that. I don’t know if you have a dedicated recovery plan for AD, but that’s one of the great things we’re gonna talk about today. Now our expert panel that we’ve got here can explain why an AD specific strategy is so important and, most importantly, give you some practical tips for how we can do that. So let’s see the results of that poll. And wow. Okay. So fifty one percent are not looking to move away ever. You’re in Active Directory and committed to it. Not surprising there. Mhmm. But there are fifty percent of you, it looks like, broke up across the different areas that are looking to move away potentially. One year, three years, or five years looking to move to the cloud. So interesting, divergence potentially in platforms there. So we’re gonna definitely, as we have this conversation, need to think about how we support a hybrid approach, that works for both cloud and on premise. So with that, I’m gonna hand it off, Sean, to you to kick off our presentation. Thanks. Thanks, John. Appreciate it. As John said, I’m obviously, my name is Sean Deuby, Principal Technologist, North America for Semperis. I’d like to say that I’ve been doing Microsoft identity as long as Microsoft has been doing identity. So back in before before Windows NT came out, up through the modern day, I spent, I was one of Intel’s original Active Directory architects and then helped run the team for ten years. So plenty of practical experience in large scale Active Directory and identity environments. And I’ve done a bunch of other things, a lot of writing about identity and speaking about identity, years as a Microsoft MVP. And I’ve sort of come around full circle back to Active Directory because it remains so critical for our modern infrastructures, even though the technology is one year short of a quarter of a century old, which is really pretty hard to wrap your head around. Uh-huh. That is. That hurts a little bit. It is. Yeah. Exactly. Yeah. That’s you can see the gray. So and this is my good friend and colleague, Alix. I’ll let you introduce yourself, Alix. Yeah. No. Thank you. And thanks to those who have joined. My name is Alix Weaver, and I’m a Senior Solution Architect here at Semperis. And I’ve worked in the Active Directory space for many years, and I was part of Intel’s Active Directory team with Sean. He was my original mentor and then went to, Nike and also the Department of Interior and FinTech. And now here I am at Semperis going on my fifth year. Right. So we’ve been around identity and Active Directory for our combined years for Active Directory. I don’t want to add up exactly Yeah. How many they are. What time? But but no. But quite quite a bit. And and, actually, John, as you were doing the introductions and you were talking about, you know, just look in the paper for the cyberattacks, I don’t know if people on this call are aware that there are there’s a ransomware attack going on right now for critical infrastructure in Lubbock, Texas. The Texas Tech Health Science Center and University Medical Center in Lubbock are days into a widespread IT outage because with UMC confirming, it is seven days into a ransomware attack. So very, very relevant, very, very here and now. I think the poll so I like to ask this question at a lot of webinars, because I think the answers are always interesting. Often more often than not, this is actually a pretty forward looking crowd here. Most of the time, then the I get never at about seventy percent. Yeah. That’s deeply concerning. Yeah. So this this shows folks that are, you know, they’re there at least some folks are actively trying to to move it. Now it’s a heavy lift to be sure, but it’s good to see that there are plans in place. Yeah. So we’re talking about identity. And we’re talking about identity and security and how Active Directory fits into that. And for starters, I think everybody is familiar by now that identity has become fundamental to modern security. Hopefully, that’s the case. If you it’s it has become formalized. So this top quote is from the NIST, formal document, about implementing a zero trust architecture. And there are multiple aspects that they talk about for implementing a zero trust architecture. But specifically, they say that enhanced identity governance, and what that really means are your identity systems with governance wrapped around them. The identity life cycle, monitoring, you know, not leaving accounts around, all of that around governance. That enhanced identity governance is seen as the foundational component of zero trust architecture. So there you have it. Right from the National Institute of Standards and Technology. And of course, Gartner, you know, says that identity is central to providing appropriate access and secure access. It’s really at the center of things. It is. And this slide I like this slide, but I also dislike it. Right? There’s a couple of reasons here. When you look at it, you know, you kinda laugh, and we all do. But the thing that’s a little frustrating to all of this is that ransom ransomware is the identity attacks are the biggest. Right? They’re the greatest risk in operational resilience. Now I know my words aren’t showing up here on this slide. Technical difficulties. So when I click there we go. Thank you. I wasn’t sure if I clicked again. So you can read here that that first step is the in a ransomware is the compromise, the identity compromise, and that’s what’s so deeply concerning. Because the very first step, that foothold into the door is that identity compromise. And according to research, eighty percent of all breaches begin with compromised identities. Right? And that’s something that we obviously wanna make sure we’re focused on is identity based solutions, right, in our space, protecting our Active Directory store. Identity based cyberattacks look like, you know, phishing, phishing, social engineering, credential stuffing, business email compromise, and silver ticket attacks. Right? All of those things can lead to a devastating impact on our Active Directory environment. So all of this points to the importance of having solutions in place that protect your identity. The last quote, the attackers don’t break in, they log in. The reason I dislike that so much is because it’s totally true. Once they have your identity, they’re going to log in, and it’s just a matter of time and probably short time before they get admin or elevate their permissions in your environment. And that we obviously don’t want that. Right? I’m gonna jump ahead to the next. And this is a great slide because I think it shows the vulnerabilities within our environment as a whole. Right? So a holistic view. We’re looking at, you know, possible IT services. We’re looking at networks. We’re looking at different attack techniques. Right? And what you’re seeing here is not only the attack techniques on the left. Right? So we know that some vulnerabilities are targeted at these rates, but what we’re seeing on the right hand side is the success. And that is what is so deeply concerning because we can see the the second one there or the third one down, excuse me, is Active Directory. Right? So we look at the network techniques, that top one. It looks like it’s highly vulnerable as well. It is. And that could be possible misconfiguration, outdated protocol, insufficient encryption, things of that nature, open ports. Right? But the chances of success are only one percent. But then you look at Active Directory, that third one down, has a high attack vector or vulnerabilities, and then the success is equally as high. And that, again, speaks to why Active Directory is so commonly targeted. Right? It is just an easy attack vector. We spoke to how Active Directory is twenty years old. And when it the advent of AD, there were certain things that didn’t even exist. Like, mimikatz wasn’t out there. The cloud wasn’t even out there. So we have to update our security, and that’s that’s really our responsibility here. And we know that Active Directory has you know, it’s it’s a highway. Right? It’s almost like a gateway for attackers. You have weak and poorly managed passwords. You have Kerberos tax, unconstrained delegation, you know, misconfigured accounts. Just general misconfigurations are so easy. GPOs, service accounts, the ban of everyone’s existence and passwords, vulnerabilities that exist, all of that. I could go on and on, but I won’t. But just that to say, Active Directory is right for the picking, and that’s why we have to really up our game. And I think that ties together in this slide is if Active Directory isn’t secure, nothing is. Because Active Directory is the core of every business. It is your identity store. It’s your source of record. Whether you’re feeding into it or out of it, it is your source of record for all your identities. Even if you have Entra, so you have that hybrid environment, everything comes out of AD. That’s that single source of record. So any impact could be or could I shouldn’t say could, will have a devastating ripple effect in your environment. That’s why we really wanna make sure that we protect it. Everything that is in your environment is also dependent on AD. You think about all your downstream applications. Right? They’re built off of or depend on AD, even your upstream applications like HR systems. Right? So Active Directory holds the keys to the kingdom. We know we need to update our security, and we really need to focus on that zero trust architecture. Right? That’s where we wanna make sure we have in place ways in which we can verify and constantly check-in real time the indicators of exposure and the vulnerabilities that are in your environment. Yeah. It’s sometimes I encounter just to be absolutely clear. When we say Entra ID, we mean Azure Active Directory. Yeah. I know the one time I got the updated term right. I still wanna say Azure. I’m still mentally doing it in my head, but I often get the question that people will ask me, what is this Entra ID of which you speak? And it’s it’s Azure Active Directory. Blame the Microsoft marketing people. Yeah. I know. It’s hard, and it’s been a hard switch. I was so proud of myself. But you’re right. I should always always say both. So this is really why deploying an identity based solution is a foundation for zero trust architecture. In most engagements, threat actors or cyber criminals, really, that’s what I wanna call them, are taking full control of your Active Directory. Directory. That’s their goal. Right? Ninety percent of attacks investigated involve Active Directory in some form of fashion, and that’s why this it’s so important to protect your AD environment. And it’s clear why cybercriminals are attacking AD. Right? It has numerous vulnerabilities. They want to get in and get your data, and they wanna cause damage. That’s unfortunately, that’s their goal, and that’s why they’re criminals. I want to inject one thing in here, Alix. Yeah. Please do. And, actually, it it’s there. And so I’ll put it in before instead of after because I jumped again on this. But since we put together this webinar, a very important report, a very, very high profile report has come out on Active Directory by an intelligence alliance called the Five Eyes. And I don’t know if you’re familiar with the Five Eyes, but those are the intelligence services of the United States, Australia, New Zealand, the UK, and Canada. The English speaking, major English speaking countries that all contribute to intelligence. They came out with a report four days ago called detecting and mitigating Active Directory compromises. And the introductory to this the introduction to this report on I’ve asked Chelsea to if, she’ll send you the URL for this. I don’t think she can post it in chat. But, you can also look for it if you search for detecting and mitigating Active Directory compromises. You’ll you’ll get it. It’s like an eighty page report. And the first couple of pages, the executive summary are basically telling you what we’re telling you about this, and then lists fifteen different major Active Directory attacks and how to defend against them. So now no one’s gonna listen to this for the next five minutes as we as they go dig up those reports. But, Chelsea will send it to you so you can you can figure out how to download it. So I apologize, Alix. I stepped in early. Go ahead. Not at all. Not at all. No. I appreciate it. I think it’s important to, disseminate all the information we have on protecting AD. Let’s see. So yeah. Sean, I think this was back to you. That is me, and I’ve just gone all blurry here too. So Oh, no. Oh, no. That would be a the ITDR, right, being a top trend. That’s my mic. So this is my shameless plug for the Hybrid Identity Protection podcast, the HIP podcast. Ah, there I came back. Where, I talk, where I talk with a number of, individuals about, Active Directory, Entra ID, hybrid identity in general, and security around identity, in general. So go check out the HIP podcast. It’s at all your finer, podcast purveyors, there. That’s my shameless plug. But specifically around Identity Threat Detection and Response, ITDR. This is a category that Gartner came up with a couple of years ago. And specifically, it is about protecting the identity infrastructure. We rightly focus on protecting credentials, protecting identities, because they are used for access in so many different ways. And continue to be used in so many different ways. But what has been neglected until recently, is recognizing the criticality of protecting the identity infrastructure because of what we’ve just talked about, how it runs the business, and how often it is attacked. And once it is attacked and once it is owned, the threat actors can pretty much do whatever they want. So Yeah. ITDR is about correct and secure operation of the identity infrastructure. And these tools discover, as Alix has already said, indicators of exposure and indicators of compromise in your identity systems, specifically Active Directory and, Entra ID. There’s confusion about where ITDR fits into the cybersecurity landscape. And so what I did is I created this slide that shows the cyber kill chain and how ITDR fits in compared to endpoint detection and response, EDR or XDR. So if you build out the cyber kill chain, what you have is initial client access. And as we have said, it occurs in a number of different ways. It could be a password spray attack to, attack, a user ID that’s using common, common password or a breach replay. Or another popular one is a VPN that has no multifactor associated with it, or some other endpoint. Phishing attacks where the victim simply gives over their credentials to the threat actor, or the old good old fashioned brute force where an attack is hammered, hammered against a set of credentials until they find the correct password. Once they’re in, you have the cyber kill chain. Local escalation on the client, reconnaissance around the network, propagation to other endpoints on the network, eventually escalation, typically an Active Directory escalation, exfiltration encryption, and finally, data extortion. And to this classic chain, we also add the escalation into your cloud service. Because about ninety percent of organizations now, are hybrid organizations that have a substantial on premises footprint, but they also have a Cloud footprint. So this complicates the escalation story. So in the beginning of this kill chain, and these are not absolutes, but in general terms. The beginning of the kill chain is the province of, endpoint detection and response. The threat actor that has phished the user is logging into the client’s endpoint, whether it’s a PC or maybe it’s a maybe it’s a jump server, that is an administrative server that is on the other side of that VPN endpoint. The threat actor logs in and they try to drop malware into it. They try to move around and do privilege escalation on it. That’s where endpoint threat to endpoint detection and response is at its strongest is finding out what’s going on in that environment. And as as they’re trying to move to the various endpoints in the environment. And then what you see in this, in the kill chain is escalation. So this is where the threat actor has moved around from client to client. They’re using something like Mimikatz, and they find that someone with administrative rights has logged on to Active Directory administrative rights, has logged on to that client a few days ago. Maybe they’re, an IT support person, something like that. And they use, they use Mimikatz to extract the credentials and the password hash of that privileged account. Bang. Game over. They now have administrative control over Active Directory, which means they have administrative control over any server that depends on Active Directory for authentication and authorization. It also means that they can then use that, they can use that tactic to move up into Entra ID Azure Active Directory as well. If they can gain control of the Azure AD Connect server, now called the Entra ID Connect server, they can do the same thing. They can escalate up into Entra ID. And in fact, that’s along the line of what happened with MGM Resorts and Scattered Spider, almost a year ago now when they gained control of MGM Resorts. So that’s the that you can see the province of endpoint detection and response and Identity Detection and Response, Identity Threat Detection and Response. And ITDR is in both on premises and for the cloud services. And cloud services is obviously typical endpoint detection and responses, not cloud service focused. Right. And one thing too to think of, Sean, is that a lot of times, Entra ID can also be a target. Right? And then that can be attacked, and then people can get back to your on prem environment. So it’s really all forms and fashion. Anywhere they can get in, they’re gonna they’re gonna try. Right? They just have to be successful once. Right. That’s right. So I put together a couple of slides in explaining this to to folks. One segment of folks in security, especially, especially security management and security strategy is really focused on threat. CISO, the office of the CISO, and those groups are focused on what are the threats in the environment and how do we mitigate those threats. So we they look at the world in terms of threat. And so the way you determine what threats you need to pay attention to, because you’ve only got so much money and there’s a lot of threats out there, is you do a risk analysis. And a classic risk analysis is basically where you determine the risk level of any particular event by looking at the vulnerability of the system, of the system that you’re focusing on, times the likelihood of a threat that is exploiting the vulnerability, times the magnitude of impact if that event actually happened. So to give you a very practical example that helps make sense of it, let’s look at data centers and power outages. If you’ve worked in IT for very long, the data centers and power outages are something that’s very important to you. So if you take that, you say, let’s look at the vulnerability to a power cut. Well, we know that the vulnerability to a power cut for a data center is now pretty low because almost every data center has mitigated that risk by putting in a battery or a diesel backup. But we take that vulnerability times the likelihood of that cut happening. Again, pretty low because power companies are paranoid about not about, you know, cutting off power to a data center. But there is there’s always a possibility there’s gonna be some monobrow with a back hole that goes crink, and then bang, you’ve got a power outage. And then times the impact of such power outage, which is, of course, very high if a data center loses power. So what you have is a you have is a low risk and a low likelihood times a very high impact. So what you get for that is a low to moderate risk. So with that in mind, let’s take a look at it in terms of Active Directory and how does that fit for the risk analysis. So the risk is around Active Directory vulnerabilities. How many Active Directory vulnerabilities are there? Times the likelihood of a cyber incident involving Active Directory vulnerabilities, times the impact of an Active Directory outage. K? Let’s plug this stuff in that we were just talking about. AD vulnerabilities, high. Most organizations have had AD in production for decades. And I describe it as, twenty four years of hurried choices. Maybe not the most secure thing that you’ve ever made. Every production Active Directory, and we have a free tool to help you evaluate it that Alix will be talking about in a minute. Every Active Directory has got a ton of vulnerabilities associated with, just the way it is. The likelihood of an incident involving Active Directory, we’ve just showed you, ninety percent. The impact of an Active Directory outage, very high. Your Active Directory is down. You can’t log in. It’s very high. And I would add one more factor to that, which is how long it takes to get Active Directory back up once it’s been down, which is also very high. So what you get is a lot of highs. High, high, very high, very high. You end up with high to very high risk. My point being, this is the sort of thing that you can present to risk management to help them understand the the risk that, an unprotected Active Directory poses in you in your environment. So with that behind us, we have our next poll question. Do you have a customized tested Active Directory recovery plan that is focused on cyber attacks? If so, when did you last test it? And I know from personal experience, there is the, oh, yes. I’ve seen the Microsoft web page about forest recovery. All the way down to, we have a customized we have a customized process that we test on a regular basis, etcetera, etcetera. There’s there’s the full gamut of that. So Yeah. We’ll see. And these are anonymous. I remember those those tests that we did years ago where we would pick someone on the team, not the author of the document to test the process. Remember that back in the day? Well, it’s very, very difficult to do AD disaster recovery tests just using native tools. It is. From a management viewpoint, it’s very expensive because it takes people and time. It does. It does. It’s so critical. So we’ve got twenty six percent are admitting their their deepest fears on, but that’s okay. We don’t we don’t see we don’t see who it is. No. And a lot of people are just like, I think I’m trying not to think about it. Okay. So we’ve sort of settled out about now. Looks like, looks like we’ve got, about all we’re gonna get. So let’s move forward. Yeah. Yeah. Got one more. In your experience, Sean, is, like, the average that you see when talking with folks. How many people are doing at least yearly, would you say? It’s gotten better. Okay. I like to hear that. It’s definitely gotten better. But, Good. What I find is human nature. Oh, we’re pretty good. We’ve got a plan. But, you know, in our business, everybody throws the Mike Tyson quote around. Everyone’s got a plan until they get hit in the face. That’s what that’s why testing it is all about. So Yeah. Alright. Well, let’s see what we’ve got here. Okay. Ten percent one month, seven three months, twelve percent six months, one year. Never tested. Don’t currently have a plan. Forty one percent. Okay. Yeah. Yeah. Don’t feel bad. You are not alone. And, of course, you could argue that, if you if you were really good on all this, then maybe you don’t necessarily even need to be in this webinar. But, that’s okay. We’re here to to move forward on this. Yeah. Okay. So let’s talk about what actually happens, in a cyber incident and how you recover from these aspects. So this is a, as you can see, a cyber attack recovery timeline, and this came from one of our largest partners from their data protection group. And this group gets called in to incidents to help the customer help their customer restore data after a cyber attack or during a cyber attack. So they at the time of when this slide was created, they had been in about a hundred different cyber attacks. And this is a very empirical based on their experience times that it takes to do things. So you have this dwell time, and dwell time is Alix and I were talking about this. Dwell time is all over the map nowadays. It’s Yeah. It can IBM says it’s two hundred days. Really aggressive groups do it in a matter of twenty four hours, but we just picked somewhere in the middle of a dwell time, time, especially of, for example, the nation state actor. And this is about this is about, utilities and critical infrastructure here. A dwell time of ninety days. They’re in there. Okay. And then there is an event that happens, whether it’s encryption or something else that comes about. Then there’s a certain amount of time that it takes to identify that something has happened. Then you have they have an average of twelve hours to invoke the incident response plan. Hey. Let me call Bob. Oh, he’s on vacation. Oh, let me call Jim. He’s out at the beach. Let me see if I can find him. Get him all in the price of the bridge, in time. And then the cyber insurer comes in. And then you have to consult with the cyber insurer, and you have to do what they say. Like, don’t start recovering while we gather forensics. Let’s get the negotiator involved. There’s a lot of a lot of a lot that goes on in that step. And then a forensic analysis to try to figure out what the heck is going on in the environment. And then you have this area. Overall, to build the minimum critical IT infrastructure, they found that it takes an average of twenty one days to do those steps. And of those twenty one days, it takes typically two weeks to recover Active Directory. So two thirds of the time of recovering an IT infrastructure, two thirds of it is just recovering Active Directory. So how many times do you think someone’s sorry, Sean. I was just gonna I have to jump in. How many times do you think a manager has gone to the AD team to ask them when AD will be up at that point? Well, having been in having been in sort of a not quite this situation, but thank goodness, but in an AD outage type situation, it’s about every minute while you’re trying while you’re trying to think. So Yeah. It definitely helps with troubleshooting skills. I just have to bring that up. So okay. So why does it take so long? What’s the situation that that requires so much work? And by the way, this this partner of ours talked about coming back to the management and going, my gosh, we had this all wrong. We didn’t we didn’t have any idea how hard it really was to bring back Active Directory. And it the way I break it down is it’s sort of in in two two segments. The first is that the servers themselves can’t be trusted. These are the domain controllers that host Active Directory. So if Active Directory has been compromised, you know, there’s gonna be malware that’s sitting on the domain controller. Who knows? Maybe they’ve disassembled, you know, disabled the endpoint detection and response software. There’s any number of things that can have happened in this. Now, this gets complicated because Active Directory backups aren’t like regular backups. They’re not like a backup of a file server, where you could go back however many months you want. Their shelf life is much shorter than that, and Alix, we’ll talk to that in a second. But the bottom line is that if you restore domain controllers with conventional backup and recovery software, they’re gonna have the malware on the domain controllers. So you’ve gotta make sure, number one, that the domain controllers are clean themselves. Were you gonna say something Alix? No. I just a hundred percent agree. I was just nodding. It’s a very difficult process, and you wanna focus on products that are just looking at Active Directory. Right? And restoring AD, just that database. Yeah. But here’s the part that most people don’t think about is that the Active Directory service itself can’t be trusted. So Mhmm. If a threat actor has gotten onto a domain controller, gotten control of Active Directory from a privileged account or whatever to drop malware on the domain controller, or to get into Active Directory, it means that they have admin rights on the entire forest on all the domain controllers, and you don’t know what they’ve done. So. Right. If we go back to the beginning of what we’ve been talking about today, Active Directory is the core of your trusted identity environment on prem and in the cloud. And if the threat actor has been in the Active Directory, suddenly you can’t trust AD anymore. And you’ve gotta rebuild trust in AD, and that is really, really difficult. Yeah. Incredibly difficult. So that’s what we’re talking about is how do you restore. Right? We know we all have backup solutions of some sort in place, and we wanna look at a restore or recover capability that, focuses on AD. Right? One of the things you really wanna focus on, like Sean had just mentioned, is just Active Directory itself. Right? The n t d s dot dit, that Active Directory database. That’s something that you wanna separate from that operating system so you can reduce the chances for reintroducing malware. Because if you don’t find out the who, when, and where someone got in and work to mitigate those risks or really eliminate now because they have been attacked, you are going to be rebuilding your AD environment in another two weeks. So you really have to remember it’s restore, but it’s also restoring to a trusted state. And that’s what we wanna focus on is how do we get back to that trusted state. So you’re gonna obviously go to the most recent backup you have. The issue here is, again, we all are doing backups in some form or fashion, but DCs are a little bit different than just a file server. So, yes, you can very easily use a number of different recovery options to do just one DC, but not all DCs at once. And when you look at your backups, you’re not gonna wanna go older than or look back further than two weeks because you have so many changes. Like, in an enterprise environment, there’s hundreds of thousands of changes that happen. This is on users, groups, passwords, computer accounts, applications, anything upstream, downstream. So you really want to be able to get back to that most recent point in time. And, additionally, Active Directory doesn’t really wanna be restored older than two weeks. It’s gonna have a lot of problems and you’re gonna spend a lot a lot of time, what I call, fixing forward. So what you wanna do is have something within that two week window. Anything else is really for forensic purposes. And then also we have Entra ID. Right? That is very complex to restore. Painful might be a better word. So we have one of our be factored in. One of our one of our experts is deep into I mean, if you think on prem Active Directory and Entra ID and the way they interoperate all the time, when you restore Active Directory or did I use this? Yeah. This side for you. When you restore Active Directory, it’s out of synchronization with Entra ID, and you’ve gotta rebuild that. And that’s complicated in itself. Very. So there’s a lot of moving pieces. Right? And that’s what we wanna make sure we focus on here is how do we restore all of those moving pieces. This I think this slide deck perfectly demonstrates or showcases all of the different components and the complexities of a restore. Right? So not only is there Active Directory and then you have the hybrid environment and, like Sean just mentioned, that sync, you have all these unique identifiers that have to be restored. And when they get out of whack, then you have all sorts of problems. Right? That’s a whole another discussion for another time. But all of this really points or showcases how many different dependencies you have on Active Directory and the complexity of restoring AD. Right? That’s what we wanna look at is how do we make sure we are gonna be able to restore AD to that trusted state. It’s not just about the data. It’s about all of your changes. It’s about all of your applications. It’s about all of your permissions. All of those things are critical to the functionality of your environment. Right? You don’t want to have to start from, you know, ground zero. So this is something that is a huge vulnerability if you don’t have that restore plan because it puts your organization, your company at risk. Right? So what we’re gonna talk about is you is what is an Active Directory Forest Recovery? Right? They put here, I’ve heard like, the first line, the boogeyman of AD operations, and it really is. It still makes me cringe when I think about it. It’s a twenty eight step, multi threaded process, and it’s a hurry up and wait game. And it’s painful to go through. You can look at the documentation, and you should definitely have it downloaded, but it’s entirely manual, fraught with the possibility of human error. It’s incredibly complex. But if you don’t already have that documentation from Microsoft, you know, please, you know, look at it and and just get an idea of what that takes, right, to do that restore that authoritative restore. If we look here, this is a little bit of an eye chart. These are questions that you wanna have answered prior to a crisis striking. Right? And this really is as much of an eye chart it is, and I say it tongue in cheek. You really should review these questions so you get an idea of what needs to happen and what you need to commit to with your different teams. The CISA, the let’s see. This is a mouthful. I wrote it down. The cybersecurity and infrastructure security agency. So it’s c i s a dot gov. They have really good templates for a starting point for tabletop exercises or even disaster recovery drills. So it gives you a good starting point, and a lot of the questions are also listed there so you can put those, you know, hopefully develop a plan. Because a lot of the restore tasks, the post restore tasks are manual. So that’s, again, where you wanna look for solutions that allow you to restore them more than one domain controller because you are not gonna wanna spend your worst moments in the best and brightest of your team members promoting DCs and rebuilding DCs. You have so many other things to do and to focus on. Alright here. This is the document that we were talking about. This is the all of the steps. Right? This is a Microsoft’s, Active Directory authoritative guide. And you can see there’s twenty eight steps here, and there’s little clocks associated with each where you have to wait. And it’s painful. I remember bringing up in an environment, bringing up a DC to do a store in a lab, and I brought up the second DC too quickly. Had to start the whole process over. I just wasted five hours. Right? And that’s in a in a test environment. So you definitely don’t wanna have that happen in real time when you are restoring a forest. And, again, this is a very labor intensive process fraught with human error. I wouldn’t wish this upon my worst enemy. So really look for solutions out there that can restore all domain controllers, not just one simultaneously, and at most importantly, restore them to that trusted state. Here, I have a lot of, hurry up and wait. Again, I think I said this. If you have multiple domains, you’re building them one at a time. It is very, very slow. And this is prior to any applications and functionality being restored in your environment. Alix, so one thing I wanted to point out in that in that previous slide let me go back. Is is the just that last little bit, and I think we have to do the animation again, is that a general purpose backup only helps you with this one step. So the twenty eight, twenty nine steps, and you’ve got a general purpose backup and recovery, it’ll help you there, but you have to do the rest yourself. Yeah. And that is it’s very slow. And I’d say like I said, it’s also fraught with human error, and that’s deeply concerning because this is not when you wanna hit, you know, all these errors and things that I mean, how long ago or how many of us remember setting up an AD Forest. Right? I mean, it’s a long time ago. So if you’re doing a rebuild, yikes, from scratch. It’s tough. So I thought I’d quickly go over an example of what it takes in what our own incident response team did to rebuild an Active Directory Forest that had been attacked, and the steps involved in it. So this is, obviously, a timeline in the front. And the top of the slide is the production network, and in the bottom the bottom of the timeline is the isolated network. So we were called into a large, telecommunications company, not in North America, to assist them because their Active Directory had been compromised by threat actors. As it turned out, four different threat actors. Groups at the same time. So they were they were in a world of hurt. So they had a compromised Active Directory on their production network with multiple threat actors in it. The first thing that our team did is they took our Active Directory Forest Recovery product and took a backup of the Active Directory forest while it was still available. And, ADFR, as we call it, is designed to back up Active Directory, but it leaves the operating system behind. And then what happens is, that means that’s left all the malware that was on the operating system behind as well. So with ADFR, they restored it, the forest to an isolated network. And then, so they had a good Active Directory backup where the domain controllers were clean, but the threat actors had still contaminated the Active Directory service itself. Again, servers and service. So the servers were clean, but the service was still not trustworthy. So at that point, the team had to do a vulnerability analysis. They looked at how hard was it going to be to clean up the production Active Directory versus take the offline one and make it clean and secure and switch it over. So they had to come up with a decision of what they’re going to do. Mhmm. The decision was to leave the production Active Directory environment untouched so the threat actors didn’t know what was happening and then clean up and strengthen the isolated network forest. So they did privileged group analysis. They implemented administrative tiering models. They, cleaned up, organizational unit permissions, cleaned up group policy objects, and more associated with making it a truly secure, a truly secure forest. At that point, they shut down all the production domain controllers, then expose the new clean Active Directory to the production environment, and then they shut down and restarted all of their systems. So when they came back up, they would go to DNS. They would look for the Active Directory SRV records, find Active Directory, and come back up, and their environment was clean. Their Active Directory was clean. The threat actors were gone from the environment. But as you can see, it’s a complicated thing to do. Also, I wanted to mention, and this is something that I we’ve been talking a lot about recently that we think is really important to be that for folks to be aware of is the vulnerabilities around your data in Entra ID, Azure Active Directory. We all as we started using Entra ID, Azure ID, we’re thinking, thinking, okay. Microsoft’s got it. It’s SaaS application. You know? We’re not worried about the data because Microsoft has got it. Well, guess what? This this slide is built from a page which you can see down at the bottom, which is called the Microsoft shared responsibility model. And, essentially, what it says, is that you, not Microsoft are responsible for protecting your own data in the Entra ID tenant. So there is a concept of soft delete for some Active Directory for some Entra ID objects, users, Office 365 groups, and, applications now that are, that are backed up, soft deleted for thirty days that you can restore them. After that, they’re hard deleted, and you cannot get them back ever again. Every other object in Entra ID, if it gets deleted inadvertently or intentionally, is gone. It’s hard deleted. It can’t be recovered. It has to be recreated. And that’s a big deal because like Active Directory on premises, these objects have unique IDs. They’re called object identifiers. And once they’re gone, it’s the equivalent of what you have on on prem, in AD, like, Accolade. Once they’re gone, they’re gone. So all of the relationships, all of the permissions, everything’s gone. So the the analogy that I like to use is imagine you’ve got a you have a car, and your car gets stolen. A threat actor steals your car. And, the police, Microsoft, recover the car for you because Microsoft will recover the tenant, and then they return it to you. But guess what? Someone has burned to the back seat and thrown garbage in the front seat. It’s not the police’s responsibility. We got your car. There you go. You’re responsible for protecting that data. You need to follow-up on this and make sure you understand your responsibilities. I’ve sort of talked about this already in hard deleted objects. Bad big deal. The operational impact, as you can see from several different examples, can be very, very large. I get the question when talking about this. Hey. Well, we populate most of Azure Active Directory. I’m still doing it. Most of our Entra ID from on premises. So it doesn’t matter. Well, actually, it matters a lot. You have, Entra ID Connect, formerly known as Azure AD Connect, and let and you say, okay. I wanna synchronize an Active Directory. I wanna synchronize this organizational unit that’s got a bunch of users in it and this organizational unit that has a bunch of computer accounts in it. Let’s just say, someone can go into Entra ID Connect and it and determine what gets synchronized and really inadvertently uncheck a checkbox because they don’t really know what they’re doing and and say go. And what it will do is stop synchronizing, for example, those computer accounts. And what will happen is they all of those computer accounts will be removed from Entra ID, and they’ll be gone forever. Even if you go, oh, no. I didn’t mean to do that, and you check it you check the box and start resynchronizing them again, guess what? They’ll be created with new organizational IDs, new OIDs, and they won’t be the same. Yeah. It’s a huge it’s a huge deal and something you need to be aware of. So Yeah. That yeah. That brings us to the key takeaways. Right? So we know that Active Directory is the Achilles’ heel for operational resilience. Right? It’s frequently attacked. It has a lot of attack vectors, and they’re successful when attacked. So it’s also hard to hard to undo changes that have happened in AD. And a lot of times, we’re not able able to spot the changes that have happened. Right? We don’t have a way of maybe monitoring in real time. Right? That’s something that definitely comes into play is what’s listening at that replication stream layer level that allows you to see all writable changes on objects in AD as it happens, seeing it as a domain controller sees it. We know that Active Directory is incredibly difficult to recovery, to recover. I hope we’ve shown case that if nothing else on this time so you can get a plan in place. Right? And recovery, again, isn’t just about recovering domain controllers. It’s about recovering and restoring to a trusted state. Right? And then we have Entra ID. We just talked about that. That’s critical. That data is critical. That unique identifier is critical. That has to be factored into your disaster recovery plan. So we have, Purple Knight, which is a great free, utility. So shameless plug here for Purple Knight. And, again, that will be sent to you. That link will be sent to you, but you can download this. It is free. You don’t have to be a domain administrator to run it. You just have to be a running it from a domain join system. It breaks down Active Directory into six different categories, so you can look for indicators of exposure in each of those categories, then work with your team to mitigate those risks. Those indicators of exposure that, you know, will be shown or will be revealed in your environment all tie into within this framework. And so they give you an overall a security posture tool of the environment, a one time read at one point in time. And it’s difficult to say how long it would take to run into your environment. It does obviously depend on size, how many VCs, whatnot. So I wouldn’t run it as, like, your highest authentication peak periods of time, maybe in off hours. Right? But just a really important call out. It’s a really, really great product, and it’s free. So you could, again, use that as a baseline. And then here, just again, we wanna make sure that you’re mitigating the risks associated with hybrid AD and then testing your plan. That’s the most important piece here is looking at your recovery solutions, how, not only are you implementing or how they are implemented, hopefully, identity focus, they include that Entra ID component, but allowing for forensics. Right? You know that you’re not gonna restore from older than two weeks, but you need to have older than two weeks because you need to find out who, when, and where they got in. You really wanna look at ITDR, those Identity Threat Protection and Response solutions, again, focusing on the identity that, are fully automated and that restore to a trusted state, and that includes your Entra ID objects. Sean, do you have anything to add on that last slide? I know I went really fast. No. No. I think I think it’s, I think it’s pretty clear and pretty straightforward. Go off and look at grab the forest recovery guide and work through it, and you really have to customize it. You have to customize it for your environment, and then you have to test it because, you know, when the chips are down and I know they make environments to be able to do that as well, but Yep. I won’t dwell because we’re almost out of time. Sorry. I jumped the slide deck. Apologies. No worries. Well, thank you, Sean and Alix. Fantastic presentation. I mean, there’s so much to unpack there. I’m really glad we have the recording because I think I know for myself, I wanna go back through. If anyone has a question, feel free to enter it into the, into the Q&A box. We’re almost out of time, but if we get questions, we will follow back up with you afterwards. So if you have a question, please go ahead and ask it. One that came from Edgar a little earlier, we could share with the audience is, he had asked when we were back on a couple slides back around slide twenty seven, Sean was walking through that example of the telecom company that was based outside of the US. And Edgar’s question had been, how long did that recovery take? And so, Sean, maybe you could share that for the wider audience. Yeah. I think I reshared it out, but I but I said we so we had, you know, we did follow the sun coverage to help these help these folks starting in Europe and working to the US. So I think we probably had four different, Active Directory experts working on it. And by then these people were of the collection were multiple, Microsoft MVPs for Active Directory of ten to fifteen years each. The other was a premier former Microsoft premier field alliance Active Directory expert now working for our team. We have a lot of those on our team. And it took about six days to to do that rebuild, do that clean, work through it. And yeah. And one of the things that comes with our Forest recovery tool is a Purple Knight post breach edition, which is specifically designed to look for indicators of attack inside Active Directory. So since that time, we’ve made a tool that makes it that allows us to speed that speed that up. Awesome. That’s that’s great information. Well, I wanna be respectful of our time commitment, so we’re gonna have to wrap it up here. As we close, I really want to thank, Sean and Alix for sharing such great insights. I will come clean that this was not a cyber attack, was an IT outage, but once a long time ago, I had an AD outage in an organization I worked at. And if you think they are exaggerating how difficult it is, they are not. It is an extremely hard process, and I think you really spoke to that very, very well. So I wanna thank the team at Semperis for supporting this event. It’s a very important conversation. Lots of folks, hundreds across the country, have been part of this to to learn how to get better. But most of all, I wanna thank each of you that joined from our audience for dedicating some of your time with this community. We look forward to seeing you again at another government technology event. Have a great day and great luck, good luck to you in putting all of these great lessons we learned into practice. Bye bye. Thank you, everyone. Thanks for your time. Yeah. Bye, folks.