Cómo recuperar Active Directory cuando cada segundo cuenta

Welcome, everyone. I'm Allison with RedmondMag, and I'm thrilled to welcome you to today's event entitled How to Recover Active Directory When Every Second Counts, sponsored by Semperis. Before we begin, I just want to cover a few housekeeping details. If you have any questions throughout the presentation, please make sure to type those into the Q and A box and we'll make sure to get those answered for you. Semperis has provided some resources which correspond with today's event, so please take a moment to check those out. They are located to the right of your audience console. Today's webcast is being recorded, so keep an eye out for a link in your email to rewatch the presentation or share with a colleague. And now I'm so thrilled to announce our speaker for today. We have the pleasure of hearing from a member of the Semperis team. With us today, we have Sean Deuby, Director of Services with Semperis. So we are in for a great event. And with that, I'll pass our time over to Sean to get us started. Thanks, Allison. And thanks everybody for, attending today. I'm gonna be talking a lot about identity and a lot about the intricacies of recovering Active Directory and what it involves. I had a little bit of trouble dialing in today. This is the the SaaS world that we live in. Everything changes. And so I'm actually on my phone, and I understand there's a little bit of a latency between my phone and my voice. So please bear with me. Hopefully, you can hear me okay. I think if you're an Active Directory person, you recognize the who am I command. This is a fun variation on it. Basically, you know, why am I speaking to you here today? You know, who am I? As I said, so I've been I like to say I've been doing Microsoft identity as long as Microsoft has been doing identity. I rolled out n t three five one around the world for Texas Instruments. I was one of the original architects of Intel's Active Directory and then ran it for ten years. Then I spent actually many years as a technology journalist and covered the rise of identity as a service, which was amazing to see at the time. Like, who would ever trust their identity to a third party? And it turns out, everybody. And then I consulted, on the Microsoft hybrid stack, and then I came back full circle to Semperis because protecting Active Directory is so critical and just never these things just never change. And it has become more important than ever to do that. So I'm very happy to be back and to be talking about this. And, I was a MVP for Microsoft most valuable professional in identity for fifteen years, until a few years ago when they kicked me off the wagon. I've got a couple of poll questions for you, and I'll let you put some information in here if you have, where you stand on this. Do you have a documented and tested, identity threat detection and response plan? So this is very identity focused. Do you have a plan that specifically is tied to identity and is specifically tied to a cyber response, a response rather to a cyber, type attack versus a natural disaster. And so, if you guys could, fill that out, we'll, get a little bit of a pulse of, a little bit of a pulse on where everything is. And, Allison is going to, let me know when we're ready to go on. She's sort of the Sven golly backstage here making things work. Let's see what we have. Ah, here we go. So we have, so about a quarter have a documented and tested ITDR plan, which is great. A lot of folks have a plan but haven't tested it or you're in the process of developing a plan and, no, we don't have a plan. So the last two, I think, are are really targeted for this is what we're targeted for here, for things for you to think about when you're developing a plan in the process or the right that you recognize that you need to, but you haven't done so far. So that's that's great. And the follow-up question for this is, do you have a dedicated AD specific backup system? This is pretty specific backup system? This is pretty easy pretty easy one. Yes. No. I'm just gonna pop over to it and let's see as people start, populating this. I think it populates live. If it doesn't populate, I'm going to go back. Doesn't look like it is. So I'll go back and, do a little more soft shoeing while you you folks hit yes or no on here. Let's see see how that does. And let's take a look again on this. So Allison's telling me to give him about a minute, so I don't want her to yell at me. I'm gonna use this as an opportunity for a drink of water. Minutes go faster when you're actually presenting, so I'm gonna try it right now and see what we have. Ah, there we go. Ah, Oh, wow. Very good. Very good. AD specific backup system. I have found that many times, and this is something for you to think about your AD, backup systems. Are they about object recovery, or are they about recovering the entire forest? Because there's quite a difference. Excuse me. So let's move forward on this. So what I'm gonna do is I'm gonna talk about the state of identity and Active Directory out around the state of identity right now and then I'm gonna go dive into the complexities. So sometime if you're having a hard time, going to sleep one evening, look up the, the NIST document implementing a zero trust architecture. And in it, you will find the statement that says enhanced identity governance is seen as the foundational component of zero trust architecture. And identity governance meaning not just your identity system, but the governance process by which identities are created, changed, and, excuse me, hopefully removed. Now Gartner says identity is central to providing appropriate access and secure access. But as it happens, even as I'm speaking, the Gartner Identity and Access Management Conference is happening over the Gaylord Texans in Grapevine, not too far from where I live. And, they're gonna be presenting on, specifically zero trust in a session tomorrow. And in that session, they have all the Gartner sessions have, very clear takeaways, things to do once you're done with this. And they separate them into Monday morning, next week, and next month. As in Monday morning, it's like go do this right away. And in the slide deck, it literally says for Monday morning, stop working on zero trust if you have not addressed identity first. So there it is. Gartner says it too. That means identity is fundamental to modern security and the zero trust process, the zero trust journey. And when we're talking about Active Directory sorry. When we're talking about identity, we're talking about Active Directory. It's the de facto identity system for almost all medium and large organizations around the world. Back in 2017, Alex Simons, who is the boss of all of this stuff, said that almost thirteen million organizations use Active Directory with almost a billion users globally. Now that was in 2017, but I can tell you as someone that has been swimming in this environment since 2019, again, and talking to a lot of organizations, the number of organizations that have migrated off of Active Directory to this day is vanishingly small. And mind you, go mark your calendars, February seventeenth, in two months, Active Directory will have been released for general availability a quarter of a century ago. A quarter of a century ago, that's it's like forever in IT terms. And yet, it's more important than it's ever been because of this hybrid world that we now live in. And the fact that it's every bit as important for on premises as it has always been. And so the bad guys know this, this, of course, and so we get credential abuse for our attacks. And as I've said before, zero trust model depends on a hybrid Active Directory integrity. And a hybrid model where you have Active Directory on prem and some cloud service provider like Okta or Entra or PingOne or G Suite, what it comes down to is, I'd say, is the math is pretty simple. Security starts with Active Directory. And it's at the center of on premises systems and hybrid systems. So if you take Active Directory in your Active Directory environment, this is probably what it looks like. And this is comes out of years of being a consultant on this. You have an HR system, and the HR system provisions new employees and terminates employees and makes other changes, and it feeds into Active Directory. Or perhaps your HR system, is consolidated with other database systems like vendor and contractor databases and other, provisioning type databases through a meta directory service or some kind of an aggregator, which then feeds into Active Directory. So that's what populates Active Directory and removes users from Active Directory. And then, of course, you have everything that depends on Active Directory on premises. Administration servers, line of business servers, backups, web servers, database servers, backups, virtualization servers, also very important when you're thinking about protecting Active Directory against cyber attacks because the threat actors go after your backups. They go after your virtualization servers. And how do you administer your backups in your virtualization servers? Right now, their folks are migrating and off, but right now, it's still mostly Active Directory. Then, as I said, we have hybrid identity where you've got a cloud identity provider. Azure Active Directory, also known as Entra ID or vice versa, AWS, Okta, PingOne, choose your cloud identity provider and your on prem Active Directory identities are synchronized with that cloud service provider, the cloud identity provider, which then gives you single sign on to all the SaaS applications we use, Office 365, Salesforce, you name it. You know, that's the source of it. And then to finally bring things back full circle, this is the hybrid identity model, clearly. And to bring things back full circle, sometimes your that cloud identity provider is also used to access HR, back to HR systems. So it's in this web. Active Directory is the core of this web of identity connections. The bad guys know this too. And that's why cyberattacks start with identity compromise. Alex, Weiner is the Director of Identity Security for Microsoft. That's, in other words, that's the security for Entra ID, for Xbox, for consumer identity. And he says, when you read about ransomware attacks, they're actually the second stage because the first stage was a campaign to get identities so that they can break into the environment. And, of course, everybody says attackers don't break in, they log in. And that's been around so long it's cliche, but it is still one hundred percent true. And you'll see it used for all sorts of cybersecurity folks all the way up to Jen Easterly, Director of the US CISO organization. And as I said, the bad guys know this and they target Active Directory. Microsoft says when Microsoft incident response is engaged, in most engagements, threat actors have taken full control of Active Directory. We have on our technical team former Microsoft Dart members. That's their incident response team. And they said, oh, yeah. It's Active Directory is always involved. Mandiant says, ninety percent of attacks involved in their they investigated for their clients involve Active Directory in some form, Whether AD is specifically targeted or is it if it's used as a means to get to their end goals. I think that this is an interesting chart and just by glancing at it, you can kind of see where I'm going. This is from a chart called the state of exposure management. And what this shows is on the left, the proportion of organizations that have exposures, vulnerabilities in these areas. So for example, most eighty eight percent of organizations have some kind of vulnerabilities susceptible to network techniques or seventy one percent for RCE vulnerabilities. Eighty two percent of orgs have, have vulnerabilities associated with Active Directory. And eighty two percent, at the same time, those exposures are responsible for, are responsible for breaches. So that's a pretty strong correlation to show vulnerabilities of Active Directory and the correlation to the breaches. In September, an intelligence group I don't know if you're familiar with this. This is an intelligence alliance called the Five Eyes. So the Five Eyes is an alliance of the United States, Canada, Great Britain, Australia, and New Zealand. And that's those intelligence communities share intelligence information. So in September, they came out with this remarkable document called Detecting and Mitigating Active Directory Compromises. It's a seventy five page report giving urgent guidance on a single product. I've not seen a report from the Five Eyes that does such focuses on one particular product and its vulnerabilities. In this report, they document seventeen different attacks against Active Directory, which, you know, you may choose or to get into or not get into, but I strongly recommend that everybody goes and they read the first two pages, the introduction. Because this is you have the Five Eyes saying exactly what I'm saying here. So don't take it from me. Take it from the Five Eyes, the importance of this. I have a link there, but the easiest thing to do, if you do a search for Five Eyes and Active Directory, it's very easy to find. You can come take a look at that. Of course, everybody's gonna go off and look at that now instead of listening to me. Now Active Directory has a number of unique aspects to it when it's being attacked and when it's being recovered. A number of these events actually bypass the event log. So you have group policy changes where you, the fact that a group policy change has changed, but you don't know what those changes have been in group policy nor do you know who actually made those changes. This makes recovering attacks attacks like focus on group policy more difficult. You have attacks like a zero log on attack where a threat actor can essentially reset and nullify the domain controller's computer account, making it easy for them to log on, which is not logged in event logs. You have an aspect of Mimikatz called v c shadow that allows the threat actor to momentarily act as a domain controller, inject changes into Active Directory, and then demote themselves again. And those changes that are injected into Active Directory are not logged in the security event log. And, of course, there's the more mundane but very, very common misconfigured Active Directory audit settings because there are a lot of settings. And it's really easy to get something wrong. And then you have something that is not being audited, that is not being written to the event log, and you may not even know that it's not been written to the event log. So very briefly, let me talk about the phases in a ransomware attack because that always help that helps inform the rest of this conversation. So, essentially, you have the the section that is known as the initial access, and there are initial access brokers that assist with this. Most of these nowadays come from identities, come from identities that are garnered in a different types. It could be a phishing email and the user, is led to a malicious website that looks like an official log on site or some other type site that they're used to seeing and they enter their credentials, thereby giving them away to the threat actors. Maybe there there was a breach. I know pretty much everybody has got a had had a breach letter, has received a breach letter by now from one of their providers or other. And where, credentials were scraped. And then what our own individual incident response people have told us is that the vast majority of these attacks come in from Internet facing devices, from VPNs or, gateway devices or servers that have, remote access exposed to the Internet that don't have two factor authentication to strengthen them. They just use these credentials and they get straight in. Once they're in, this is where they compromise the endpoints. And this is where endpoint, this is where endpoint detection comes in. They download the malware. They connect back to their command and control systems, and they have a remote access connection. Then this is where Active Directory becomes involved. From the client, they will go and enumerate the vulnerabilities in the Active Directory that they're targeting. How do I gain domain dominance in Active Directory? As I said, from the Five Eyes report, you'll see there are many different ways that this can be done. For example, keberoasting or, another is by doing lateral movement to other systems and then credential harvesting of credentials in memory, perhaps administration administrative credentials which then gives them domain dominance, once they have domain dominance they have the ability to control any system that depends on Active Directory which is frankly most systems in an on premises environment. So they can get to the databases. They can get administrative access to the databases. They can extract data from the databases. And then once they've exfiltrated that data, they can encrypt your network. So they can extort you for both the data that they've encrypted that they've extort that they've extracted and that for decryption keys of the environment that they have encrypted, double extortion. And in passing, it's worth mentioning that we have seen through our ransomware surveys that the, decryption keys, the QA, the quality assurance on the decryption keys is not nearly as good as it is on the encryption. So a number of decryption keys have actually failed to work even though they were supposed to work. So I've talked a lot about the the challenges around this and the vulnerability of Active Directory. But when you're talking to management, senior management, management thinks in terms of risk. And how does that work in risk? So one of the this is just a very brief primer on doing a risk analysis and how does it map to Active Directory. So you can talk to management and present the risk to them in a way that they understand. So at its core, a risk analysis is simply the vulnerability of a system times the likelihood of a threat exploiting the vulnerability times the magnitude of the impact from that threat. Now we actually do a risk analysis all the time as we drive from one place to another and put on a seat belt. When COVID went on, you know, what kind of an environment do you want walk into? What is the risk associated with it? But a common example is a data center and a data center power outage risk because we all know about data centers. So if you plug a data center into this formula, what you get is, if you plug a power outage to the data center, what you get is vulnerability to a power cut. What is the vulnerability to a power cut? It's actually pretty low because this, risk has been mitigated with a battery backup because we know that power problems happen to data centers or a diesel backup times the likelihood of the power cut, which is generally low because the power providers have done what they can to make sure that the power flows to data centers very, very, highly available, so that they don't get outages. But the impact of a power outage is very high should that happen. So what you have is low and low and very high, which translates to low to moderate risk. So let's take that formula and plug that into Active Directory. So for an Active Directory risk analysis, what you get is the risk is equal to the number of Active Directory vulnerabilities times the likelihood of a cyber incident involving these vulnerabilities times the impact of an AD outage. You can see where this is going. So the risk of, the number of Active Directory vulnerabilities are high. Now if you run Purple Knight, our free community tool, you'll already have an idea of the kind of the number of vulnerabilities that you have in your Active Directory environment. And you probably have a lot. Everybody has a lot. Don't feel bad. That's just been our experience. More than thirty thousand organizations have downloaded Purple Knight, and pretty much everybody gets a bad score. So the vulnerabilities are high. The likelihood of an incident involving Active Directory, we've already shown it as high. Just ask Microsoft or Mandiant, ninety percent plus times the impact of an Active Directory outage. Very high. We've talked about Active Directory being at the heart of all of this. You can see how high the impact of it would be. And I've added one more factor into it, which I'm going to dive into in more detail here, which is the average recovery time for Active Directory. So it's not just the fact that AD is out. It's the fact for how long it is out, how long it takes to bring it back and make it reliable again. It's very long. And so that becomes very high. So what you have to present to management is high and high in two very highs. So that means high to very high risk. So in management's dashboard of risks that they need to be looking at, green, yellow, red, Active Directory's risk against cyber attack is a red for sure. So what actually happens to what we call hybrid Active Directory, which is AD plus Entra ID or it could be ping one or Okta, whatever, in a cyber incident? This is a slide from one of our providers that has done more than a hundred been involved in more than a hundred incident responses. This is a data recovery, team for one of our partners. And what they found is that empirically from across all of these one hundred incidents that it takes, excuse me, that it takes about twenty one days to recover basic IT functionality, not extended IT functionality, but just basic IT functionality in, a compromised environment. And I won't go through the details of these individual steps, but the the core aspect to point out is that they found that the rebuilding of Active Directory functionality took anywhere between five and fourteen days. And remember, with AD down, your applications can't come back up and you usually don't even begin or finish restoring your applications until Active Directory is back up. So that means out of a twenty one day average that your systems are down, that fully two thirds of that time can be taken up just recovering Active Directory. That's a really huge step. And you can see, like, why is it why does it take AD so long versus all of the other ones? Well, the way I characterize it is the result of an attack on Active Directory falls into two categories. The first is that the servers themselves can't be trusted because they almost certainly have malware on them. Threat actors drop malware on all these environments. And a peculiarity of Active Directory, and I'll talk about this a little bit more, is that you can't use backups from very far back in AD for a number of reasons. It's a very complicated environment made more complicated by the hybrid environment. Essentially, what that means is that if you recover your domain controllers with a useful backup on a recent useful backup, they will have malware on them. So you have to figure out what the malware is and how to get the malware off of them. The second is that even if you get the malware off of the domain controller, you still can't trust the service because the bad guys got in, the threat actors got in. If they were able to install malware on a domain controller, that means they had administrative rights in Active Directory, which means they were able to make changes in Active Directory across the entire forest, and you don't know what those changes are. If you simply recover domain controllers and clean the malware off the domain controllers, the threat actors have probably installed some kind of persistence which allows them to come right back in and recompromize your environment again. Now to the point of Active Directory backups in their short life, which is really not I believe is not really well understood. So if you have an Active Directory forest of any size, there's an awful lot of changes going in this environment every day, every hour, every minute. Enterprise ones, I know a large enterprise that we protect, they have said that they have approximately forty thousand changes in their Active Directory environment on a daily basis. When we install our Directory Services Protector, product in an environment, oftentimes our customer will discover that they've had an application that makes AD changes and it's poorly written and it actually is making thousands of Active Directory changes and clogging up their database and slowing things down, and they didn't even they weren't even aware that it was there. But typically, what you see are changes to users, groups, and computers, password changes, trust passwords changes, application partitions changes. And under the covers, all sorts of configuration changes, change on a regular basis. So what that means is that the older the backup, the more that you have to go back and try to remember what happened and recreate what was built since the time you recover from the backups. Very difficult even for organizations that have good governance. And in my experience, a lot of organizations don't have very good governance on what is happening, when they create users, computers, and groups. Further complicating that fact is that we all live in a hybrid environment and you have Active Directory and Entra ID. That synchronization, if you think of Entra ID and Active Directory as a synergistic whole, there's a lot going on under the covers to make those two things work. And if you suddenly bring back Active Directory, without you really thinking about the implications for Entra ID, you can really run into a lot of trouble. It's very complicated. What that boils down to basically is that you really want to be able to use backups. We say about two weeks or newer of age. It's worth asking the question, well, this is so hard to work with. What about rebuilding Active Directory from scratch? And if you just look at this picture, and I'm gonna go into this in a little bit more detail in a bit, you can see why you don't wanna necessarily redo this from scratch. It isn't just about the data in Active Directory. It's about the servers and the application permissions that depend on Active Directory for access. So it's about permissions. It's about ACES, access control entries, or ACLs, access control lists. And they all depend on Active Directory. If you look on the right, you can see that we have SID twelve forty seven and SID sixteen o four. Twelve forty seven, let's say that's for an individual user and secure SID standing for security identifier, SID sixteen o four for a group. And you use those SIDs to populate applications on premises, whether it's a business application or a SQL database or a file server. Those permissions, those security identifiers are shot all through those applications, thousands of them on a typical server or even on your PC for you to go look in the permissions on your PC. Take that and multiply it by thousands of servers and applications. You come up with millions of permissions. If you throw this away, if you throw Active Directory away and you rebuild it, all those permissions are lost. This also affects Entra ID synchronization. So it can really can be a corporate a potential corporate extinction event if you have no way to recover Active Directory and you're thinking of the need to rebuild it. Not something you wanna get into. So this brings up the topic of Active Directory Forest Recovery. I think of it as, the boogeyman of Active Directory operations. And I say that as a long time AD administrator. I would joke that's what's kept the Active Directory administrators up at night for the last quarter of a century now. It's a twenty eight step multi threaded rebuild process. And what it does at its core is it strips away pretty much everything in Active Directory down to its very, very minimal essence. And then you run a bunch of reset, operations on it, and then you have to painstakingly rebuild it out to the way you had it before. The Microsoft documentation is not specific to your environment. It's the high level steps of what has to be done. And even the high level steps, I did this operation and I took the web pages. It's only on web pages now. I took this operation and I extracted all the web pages to Adobe PDF documents and I got rid of all the extra the pages or just links to other things just so I had the meat of it. And it came down to the core processes being forty nine pages long and this the references, the appendices if you will, is being almost a hundred pages long. So the reference is being like, oh, how do you perform an authoritative restore of SysPho, for example. And then I took that document and I went through it in pen and manually added annotations to say, okay. In a cyber attack, have you thought of this? The document doesn't give you the aspect of have you backed this up? Have you taken care of this? Have you thought of that? Have you thought of how long it takes to accomplish this? And we have actually made a document out of that. One of my colleagues calls it the Guide To The Guide. My guide of things to think about to the Active Directory Forest Recovery Guide. So you have something one to look at the other and make sure that you have built a solid forest recovery process. And that'll actually be made available after it's brand new. It'll be made available after this, after this webinar. It will come out, shortly. And you'll get a link to it. The process is entirely manual. There's only one step that and as you I'll show you in a second that is, that is has any kind of automation out of the box. And it's very complicated. It's very complicated and the thing that you can't underestimate in this complication is that you're in the middle of a cyber attack. And maybe your phones don't work, and maybe your communications don't work, and maybe the documentation that you had written down for recovery is stored somewhere in a file server that you can't access because Active Directory is down. This is for those of you that are AD administrators, and I'm not gonna go through this. This is a list of questions that I cooked up, pretty quickly actually, and there are many more questions. But these are the questions that you already wanna have documented before you get a phone call at two AM on a Sunday morning from your operations center saying, hey. I can't access a number of our domain controllers. Things that you have to be prepared for in a cyber crisis. And the actual forest recovery process, I have squished down onto one page here. And, again, this is obviously very much the high level process, twenty eight twenty nine steps. And if you look on the left, you'll see that for every domain in your environment, you have to perform steps three through ten. And for every domain controller in your environment, you have to perform steps twenty three through twenty six. And every place that I have a little clock is a step that takes an unusual amount of time. You can't just crank through this in a hurry. Some of these steps take, quite a while to do and there's not much you can do about it to speed it up. For example, the unhost and rehosting of, global catalogs in Active Directory. And it doesn't matter if the CIO is calling you every fifteen minutes on your cell phone asking you how it's going. This has to happen at its own pace. So what you can see is that there are a lot of places to make mistakes. And in many of these cases, if you make a mistake at one point, you may have to go back. How do they say in Monopoly? Don't pass, go back to go. I can't remember. It's been too long since I played Monopoly. But, anyway, you may have to start over again. And that the required staff is needs are high. And this is when staff is being used to try to recover everything else. I say it's basically everybody that can spell AD has to be on deck to help rebuild the main controllers and help perform operations to get to get AD back because you can't get anything else back until AD has come back. And that one step that I talked about, general purpose backup only automates step three. So when you're doing a non authoritative restore of the first domain controller in every domain, you can use your general purpose backup to restore that one domain controller. Then you've got to go look for if you see step five underneath there that takes time, then you got to go find the malware on every one of those domain controllers and make sure that they're cleared off. Now how does this happen in real life? So this is an example I'm going to share of our incident response team actually going in and rescuing an organization that had had their Active Directory crippled by, a threat actor, actually multiple threat actors in their environment. So you'll see on the top of this timeline is a compromised network and then the bottom, which is the production network. And in the bottom is, the isolated network that the recovery work was done in. So we have your compromised Active Directory in the existing environment. So the first thing that our incident response team did was they used our Active Directory Forest Recovery product to take a backup of Active Directory. So we had a good backup of it in case the threat actors did crypto lock that environment, we still could recover Active Directory. We had a good we have a good backup of it. And the way the way ADFR backs up Active Directory, it leaves the operating system behind and it leaves all the malware that was sitting on the operating system behind and just carries forward the Active Directory role itself. Then our team recovered the Active Directory backup onto fresh servers, virtual machines in an isolated network. So they are fresh machines, so the new servers were clean, the operating system was clean. So we know we have with that point, we have clean domain controllers on the recovered Active Directory. But the objects are still contaminated. The AD objects are still contaminated because the threat actors have gotten into the service and made changes. So our team had to do a vulnerability analysis. How bad was it? How bad was their production one? What should they do? Should they try to clean up the the production one or take another route? Well, the decision was made to clean up and and harden the Active Directory in the isolated network while continuing the use of the compromised AD so the threat actors didn't know that this remediation was going on inside the isolated network. So in the remediation, our team built in. They cleaned up the privileged groups. They implemented a basic tiering model. They cleaned up OU permissions. They worked on errors and configuration of the group policy objects so that they couldn't be so easily compromised. And a few more things. It wasn't perfect, but it was a lot better than the existing environment. Then at that point, they shut down all the production domain controllers. They opened the firewall of this isolated network. So they had the new Active Directory out there and the stuff that had to be taken, so there it is. It's brought back into production, but the clients in the environment didn't know about it yet. So what they had to do was restart all of the systems so they could safely connect to the new Active Directory. They shut down. When they rebooted, they contacted DNS. The Active Directory that was clean brought back to the production was registered in the production DNS. So when the clients came up, they said, find me my nearest domain controller. They collected those DNS records, the SRV records, and they reached the new domain controllers and voila, they are up in the new and clean environment. But as you can see, not trivial by any stretch of the imagination. There's also risks from Entra ID. This is a whole other area and unfortunately, I don't have time to really go into it in great detail, but this is about the complication of hybrid identities. And this is a hybrid entities world that we live in today where you've got Active Directory and the Entra tenant. And as I said, we have these SIDs. We've got, you know, SID twelve forty seven, SID sixteen o four for an individual user or a group, and they were pushed into these on premises applications, the permissions, allow, deny, that sort of thing. Then these identities are synchronized up into Entra. And in Entra, they are given, an object ID, an OID similar to a SID, synchronized and kept in synchronization with on premises through what's called an immutable ID. And then those object IDs are part of the OAuth OpenID Connect token. The OAuth token that then goes into the permissions for the SaaS applications like Salesforce or OneDrive. Very, very similar analogous to what's going on, on premises. Well, the challenge is, is what happens if something goes on into these identities if something breaks. So imagine, for example, if, an administrator accidentally, deletes a user in an Active Directory forest and they go, oh, shoot. I didn't mean to delete that account. But when that delete happens, the operation synchronizes up into Entra and the user is deleted out of Entra entirely. Then the administrator goes, oh, shoot. Let me add that user back in again. And they add that user back in again. And what actually happens is a brand new account is created in Entra. A new user or a new, group or whatever is created in Entra. And it doesn't match what was set for the permissions. So those permissions are not working anymore, and they have to all be entirely repermissioned. So these are complications that I think a lot of people haven't really thought all of their way through and it has to do with recovery for this as well. Now, Entra, and again this I don't believe this is well known, you are responsible for your data in the Entra tenant. Microsoft is responsible for the configuration of it and then and maintaining the framework. But as you can see here as what I have highlighted, the documentation of the configuration of your environment and the data retention for your users and computers and much of what you have in there is your responsibility. And do you see also there, this is about the restoring the prior configuration. So there's no magic here for restoring data that has been deleted. There's a there is a, for a limited number of objects, there's what is called a soft delete, which is makes it good for about thirty days for users, groups, Office three sixty five groups only, applications, and security principles. Everything else is hard deleted. So what you see in Entra is this is what most of us see on the surface. They see users and groups and applications and conditional access policies, some of these principles. But what actually has what Entra has in the underneath are a ton of configurations to keep track of. So you can restore a user, for example, or perhaps a service principle, but if something has been hard deleted, which is completely gone and forever, those configurations have to be recovered. And usually, it's recovered manually. So on this page, and you can see the link below, Microsoft specifically calls out the customer responsibility for what they need to be prepared for. Not that Microsoft gives very many tools to do it, mind you, but you are still responsible for it. The analogy I use is, if your car gets stolen, the police will return your car, but they're not responsible for the state that they find the car in. That's up to you to to deal with. And most are not protected. If an object is hard deleted, it's nonrecoverable. If you create if you synchronize a group in Active Directory up to Entra ID, that creates an Entra ID security group. Security groups are hard deleted. So if you take a group on prem and Active Directory, you synchronize it with Entra, and then you use that synchronized group up in Entra ID to access a whole bunch of permissions, if that group goes away accidentally or intentionally or moves outside of the synchronization scope, all of that is deleted and it cannot be recovered again because it's been hard deleted. I'll skip this just to say that this can have significant user impact and the things that you need to be aware of. I have some actions for you to take to help protect yourself on it, but it is an unresolved situation to be completely transparent about it. So takeaways. I think you get the idea by now that Active Directory is the Achilles' heel of IT operational resilience. It's always attacked. It's difficult to undo changes, and it's extraordinarily difficult to recover. It's not just recovering the domain controllers, but it's restoring trust in the service itself. And takeaway is that your Entra ID is at risk. So you need to start documenting your configurations so that it even if you have to manually re rebuild it, at least you know what they are. What you can do to help understand your vulnerabilities is Purple Knight. So that Five Eyes document that I told you about and maybe you went out to pull down on the side, they mentioned three tools that you can use to assess your Active Directory vulnerabilities, and Purple Knight is one of them. This says twenty thousand downloads. We have more than thirty thousand downloads of Purple Knight right now. And it is a security assessment tool that has a hundred and fifty different indicators between on prem and Entra ID to show you your security vulnerabilities, where they fit in the MITRE ATTACK framework, their criticality, and to give you steps to remediate them. Very, very important. The best as my sensei would say over my other shoulder there, my, the best block is to not be there. Make yourself less vulnerable because the threat actors you read a lot about this in the press, all these novel attacks. But the reality is most threat actors, especially cybercrime, they do the same techniques again and again and again. They're not interested in novel. They just wanna get the money. So protect yourself against the basic vulnerabilities and that really helps. Another free community tool that we offer is called Forest Druid. And Forest Druid is an attack path analysis tool that helps you define what your most important, components of your directory are, what are called tier zero assets, your domain controllers, your privileged accounts, your groups that hold those privileged accounts, and it maps the path out into those privileged accounts as to find out how can bad guys get into your environment and show you some probably pretty large holes that you can close-up that you may not have ever been aware of before you ran Forest Druid. So what are some of the things that you need to be doing for protecting your environment? Number one, mitigate the risk to your recovery aspects. Look at Purple Knight and Forest Druid to start fixing your vulnerabilities. Make yourself less of a target. I talked about the Forest Recovery Procedure. Dive into the Forest Recovery Procedure. Look at go out to the Microsoft site and find it. And you'll see I've also mentioned my guide to the guide for the forest recovery process will be coming out in the next few days. Look at the ability to get forensics and threat actor detection and ejection capabilities in a product like Directory Services Protector. DSP sees all of the changes that are going on in Active Directory because we tie into the replication engine of Active Directory. So it's untamperable. We see everything, and we give you the ability to roll those changes back with the click of a button or automatically. Start looking into EntraSync recovery, and I have some tips for that. And look at Active Directory Forest Recovery, our forest recovery product, because it cuts down the time to recovery by ninety percent. And this is at a time when literally time is money. Models have shown that ... a thousand dollars an hour in larger organizations, which is crazy. And that really is the essence of time is money. And then for the Entra side of things, there is a free tool, offered on GitHub from Microsoft called Entra Exporter. Use Entra Exporter to document your configuration. I should note that it's Entra Exporter. It does not restore. Restoring is far more difficult. We offer a product called Disaster Recovery for Entra ID (DRET) that we are continually developing and making more and more of Entra ID recoverable right now to protect your Entra ID objects. And there we are. I'm at the top, and, I've managed to get through what I wanted to share with you on time. And, I see we have a few questions here. Is that right, Allison? Yes. I just wanna say thank you so much, Sean, for that informative presentation. It was absolutely outstanding. We do have some questions in from the audience, and I want to remind the audience at this time, if you do have any questions, please feel free to type them in now, and we'll get them answered for you by Sean. So, Sean, I'm gonna start off with our number one question right now. It says, do we really need a dedicated AD backup system? Well, hopefully, in my diving into detail of forest recovery, you see the number of steps that are involved in there. And so most conventional backup, most conventional backup systems, and they're coming to recognize this now, are not adequate for Active Directory recovery, not for forest recovery. Many of them do object recovery, which is great if you accidentally delete a user or a group or some other type situation like that. But if your environment has been cyber attacked, they're really not built for that. And we're actually partnering, for example, with Cohesity, because they recognize the fact that this is complimentary to a regular backup solution. Thank you. That was amazing. We also have another one from Andrew that says, what else can you recommend to a business in the context of identity protection? Let me I'm trying to look at it so I can see it again. This would be our number two question. It says, what else you recommend to businesses? Yep. In the context of identity protection. I'm working through the box there. You know, it is identity protection, and this is becoming more and more of the focus here is what are the aspects to it. It is complimentary. So identity protection in the context of identity protection, one aspect of it is privileged identity management, clearly. You're trying to wrap as many privileges because that's what you're protecting are the privileges, the privileged identities. Those are the quickest hop to get up to control of the environment. And there's obviously a very robust PIM market right now. And that's one aspect of it is you wanna protect them with the passwords they're changing and all of that. PIM does not protect you from a threat actor going in the environment and collecting, collecting privileged identities out of memory. The core of it is how do you restrict the use of that so that it can't jump in and compromise the environment. So that's my take on it. Yeah. There are plenty of things. And, of course, you know, we there's all of the work that's going on in passkeys and, passwordless authentication right now to protect you to protect the initial access aspect of it. ITDR really focuses on the identity systems themselves. Our next question comes from Bill. Bill would like to know what are the primary indicators of compromise that signal a threat actor is targeting hybrid Active Directory environments, and how can they be detected in real time? Well, gee, Bill. There are a lot of them. And, like I said, the Five Eyes report mentioned seventeen of them, right off the bat. For example, keberoasting. Keberoasting, if you're unfamiliar with the technique where a threat actor can enumerate service principles in Active Directory and correlate them to service accounts that are usually easy to recognize in service accounts and then use them to there's a process that allows you to take them, offline and crack the passwords for it. And if that service account has got a privilege that has, privileges in Active Directory as many service accounts do, bang, the bad guy has privileged access to Active Directory. But, yes, there are there are plenty of other ones. I would say that the indicators of compromise, the easiest way is to simply run Purple Knight. It doesn't require any rights in your Active Directory forest. It doesn't even require any rights in your, on the client it runs on. If you are associated with security, though, I would probably wanna tell the stock about it. Otherwise, you're probably gonna be getting an urgent phone call, that your someone has compromised your environment, someone's compromised your PC. We do have time for one last question. I do just wanna mention that if your question is not answered at this time, I'm sure Sean will get back to you to answer the rest of the questions. We do have a ton that just came in. Our next question I was going to ask is from Andre. Andre says, does Semperis work with channel partners or to, direct to end users? We work with we're a hundred percent channel, and we're with a lot of the big channel partners. Feel free to contact us. I'm sure we have some contact information somewhere and we can find out what, our partners map with the partners that you work with. So, yeah, we're we say and it's literally a part of our corporate, our corporate statement is we're a force for good. And so we really are that's why we provide these free tools to help organizations protect themselves from this rampant cybercrime environment that we find in. Also, nowadays, you know, critical infrastructure, critical infrastructure attacks. How can we help you protect your critical infrastructure from what we know, what Jen Easterly just said, in a in a interview a couple days that in her whole career, she is most concerned about what China is doing nowadays through our US infrastructure. Well, thank you so much, Sean. It looks like that's all the time we have for today, and I just wanna say thank you so much for being here and giving this presentation. I also wanna take this time to thank our audience for attending today's webcast. And of course, a special thank you to Semperis for sponsoring today's event presented by RedmondMag. Thank you again for attending, and have a wonderful rest of your day. Thanks, everybody.