Seguridad y recuperación de AD: Preguntas que necesitan respuesta

My name is James Ravenel. I have been in this industry—in IT—for almost 35 years now. And before you say, how can that be? Trust me, I had to put some Ben Gay on my back before I got here today because I'm in a little bit of pain. Chris—I wanna thank him because he mentioned something that I thought was really cool when he talked about Ferris Bueller. I remember being a young preteen that was looking at that with fascination, how he was able to hack and do all of these cool things. So Chris brought up a lot of things that I'm actually going to dig a little bit into today. How many of you are familiar with Active Directory? Just raise your hand. Yes. Great. Great. I love that. That means that, you know, we are a niche of people who know about a thing that most people do not care about. So let's get into this conversation. What are we here to do? I wanna really quickly get into the conversation. And what we're gonna be discussing is AD—and understanding that it's a critical part of your organization. We're gonna be talking about the manual process of having to recover AD—and I'll explain to you why we're even talking about that. We're gonna discuss the risk and constraints of doing that manual process, the readiness questions—that's the QTNA that I'm talking about. And then more importantly for me, I always like to remind the folks that I'm speaking to that you matter. All of the people in this room have a have a place in securing your organizations, and even securing your own private information. So I'm gonna also talk about people, process, and technology. So let's get into this. So why are we talking about AD security and recovery? Identity systems are top targets. You heard Chris in the last session. He got to this one point in one of his last screens when he talked about AD and I think he called it a joke or something like that. AD is very—out of the box—very insecure. A lot of the configurations that are included in AD don't really make a lot of sense in 2025. I consider myself a young dinosaur. And what does that mean? That means I was around and I worked for a company that helped to deploy Active Directory. It was me and AD. We were the two that deployed it for a large company that some of you may know called Verizon. And when we were doing that, I promise you we had no idea that AD would still be around after all of these years. This is in 1999, 2000—around that period. AD is over 25 years old. That means that if it was a human, it could legally rent a car now. Forget about just drinking, it could actually do something that is a benefit to you. So when AD is coopted, the thing that you need to know is that speed is of the essence in recovering that identity provider. And so is trusting that what you have recovered is free of threat actors. Let me pop over here. So about a year-and-change ago, Five Eyes—if you're not familiar with them, it's a collective of these countries that are listed up here. And they came out with these statements talking about how important AD is and why it needs to be secured and that it is a high-value target for threat actors. You heard Chris mentioning that. Right? He talked about…what did he say? 2.4 billion identities. You heard a theme in his discussion. Identity, identity, identity. He talked about endpoints and other things as well, but identity for most organizations is the key to your castle. And if you don't secure identity then you will have a lot of problems for your company beyond like what us nerds used to think about was just the technical stuff. But now, companies are going bankrupt. They're going out of business because they didn't do the things that are necessary to protect identity. So I like to call this “Can it be that AD was all so simple then?” When we were first deploying Active Directory, I was coming from an environment using Banyan Vines. Anybody ever hear of that? Novell networks, you guys are old, I see a lot of gray beards out there. So we were coming off of different network operating systems that we thought were a lot more functional, but I don't know, we made this decision to go and use AD. And it was simple back then and we didn't have a huge amount of architecture. In fact, back then we thought that this was a lot. Like, wow, you know, I've got my web server. Yay, me. Running IIS and, you know, I have my administration and, you know, my backup server and we had everything that was pretty much kinda contained. And then something happened. AD stuck around and you know, after 25 plus years, it started to look a little bit more like this. Where now, AD is getting its information from like your HR systems that are SaaS systems. And you're like, well, how can they be getting it from there? Well, this is a cyclical thing and it's this is why it's a problem. So now, instead of sometimes sending it directly to AD, now we're sending it to other meta directories and then some of your vendors and contract databases and other databases and we're starting to pull this information and then it gets into AD, our original infrastructure and then what are we doing? Now we're sending that out to the cloud again just to allow you to use authentication services to other services that you need. This thing got really complicated. I didn't have this problem when I was using Banyan Vines. We didn't think about these things. We didn't think about this with AD. For a reminder: AD is inherently insecure. So, at some point, we have to realize that we've made a glorious mess with this. And it starts to feel like we're playing a game of Jenga. And that if something happens with AD, which is the underlying identity provider for all of these systems, what happens? I see some smiles in here because you know what happens when you pull that last Jenga piece. Right? You're ready for the whole table to collapse. So what does an Active Directory recovery look like? And I'm gonna tell you—have any of you ever had to recover Active Directory? I don't see you smiling. You didn't smile when you raised your hand. It's not a lot of fun, right? Lengthy process. Yeah, it's a very lengthy process. Right? It's like a 150 pages. I don't remember the last time I read a 150-page book on something that was fun. These days, if I'm reading it's some manual that I had to. You know they say RTFM, you had to learn how to read the things so that you can do the thing right. This 150 pages is approximately 30 steps. They're manual steps and I promise you, every time I've had to do it, I make some mistake. And if you make the critical mistake, you have to start the process all over again. That is no fun. So, this is what it looks like enumerated. If you go to the Microsoft web page, they're gonna start you at…let me show you here. At this section, where they talk about a non-authoritative restore of the first writable DC. Now, we're all smart people here. I can promise you that if you had enough time, you can actually get through this process. But as—what's your name sir? Trey. As Trey mentioned, this is a very lengthy process and again, most of us in here are technical people so we don't—we're in a mode, I'd say like the last 10 years or so, we're starting to talk about business continuity, identity resiliency. These are the things that we used to talk about. We were just the nerds and we just did the things that nerds do. But now, we're always having conversations about your organization's reputation and how much money IT could be potentially losing or your security organization could be losing because you don't have things back online quickly. So anything that takes a long time to do, like this process, can be problematic. So Microsoft's process is telling you to do all of these manual steps. You'll see these little circles next to some of these, those are actually little clocks. And those little clocks are indicators of things that take a long time to do. And, I just wanna read a couple of them to you. If you see here, that first one says—number three on here—says “the restore.” And then you have number four is another restore of your sysvol. And then number five, remediate malware. If you are down because somebody in your team made a mistake—and I'm not gonna say I've ever done that, writing some PowerShell that might have done some bad things, but I have. But, if you know for certain that you don't know what happened and you could have some malware, some ransomware, maybe you got your domain controllers have been encrypted. That's a step that you have to figure out because Microsoft, if you look in their documentation and you read the SLA, what they're gonna tell you is that, hey, sorry, we're not here to help you with your ransomware or malware attack. What we're going to do is assume that you have hardened your environment already so that this never happened in the first place. So, what happens if you haven't done that? Raise your hand if you are in operations and you get a lot of alerts about things that are happening in your environment. Two, three, couple folks in here. Do you get to every single one? Because I know in my last job when I was at Active Directory Operations Engineer, the day that I left that company, I had 32,000 unread emails that were all alerts that I just never got to. That's scary. Right? We don't always get to the things that are important. So this process, and I'm not going to go through every step here, but trust me when I say—and trust mister Trey in the back— that this process is very challenging to do in a manual way. And if you had to do it, it would take days to weeks to complete. So before you commence a Active Directory recovery, there are some considerations that you must make. I'm gonna go through some of the sections here. So here's where I talk about governance, communications, and compliance. If your company right now called you and said, hey, our entire network is down—You'd be lucky to get that phone call because communicating with each other to coordinate in a cyber event is very difficult. Why? A lot of these rely on Active Directory as the underlying way for you to connect to it. Raise your hand if you use Teams in here. Anyone? Right? So, I know that when I speak to my teammates on a daily basis, I'm usually using Teams. But guess what? If your Active Directory is down or your Entra ID is down, you might be in trouble. Right? So hopefully, you have some other way, you know, you got those numbers stashed somewhere on your phone, you got a phone book of all the important people somewhere underneath a pillow or something. Wherever you're doing, I hope that you have a way to keep in contact with your team. If you have been breached, are there any regulatory requirements that you have to be beholden to? Do you have to call your insurer? Right? These are things that are important for you to be thinking about before an incident. You know, will any forensics be required? Those are—I'm not gonna go through every question here because I know we only have about a half an hour. Am I doing good on time and the back guys? Great. But, your recovery plan and procedural readiness. Under that, I'm gonna ask you, do you have a recovery plan? And if you have one, is it a plan that that has been read? Do you understand that procedure? Have you read Microsoft's recovery plan? I showed you that that was 150 pages. Right? Have you ever heard of this? Yes, you. Have you heard of that? No? This is a real thing that I can tell you when I was in AD ops, I didn't really know about this plan until it was time to do a disaster recovery event and my team called and they said, hey, you have to do this section with our disaster recovery team and when I looked at this document I almost cried. So, it's not enough to just know that these things exist—hopefully you do know it exists—but have you read it, do you understand it and how do you make it make sense for your organization? Because if you've been in this industry long enough and you've worked for more than one company, I promise that every company implements AD a bit differently. You may have some similarities but overall, they're never really done exactly the same way. And again, I'm gonna ask, have you tried whatever your recovery plan is, if you have one? Assuming you have one, have you tried to actually do the recovery? And a lot of times? People say, well, I do tabletops and that's cool. If you know that that tabletop has addressed all of the gaps that usually come up when you're actually trying to do a tabletop. Trey, when you had to be involved in that process, were there some things that you uncovered that you realized, oh shoot, we didn't think about that. Did that ever come up? Exactly. Right. So there's always something, like if you did a tabletop, you know, if you have to do this live, to recover your Active Directory, that means that your business is down. That means that no one in your company can access their computers. They can't log in to anything. That means that someone at your company is calling you saying, hey, when are we getting back up? Because why? Our business is down, we're losing money. It's probably two, three, four in the morning, someone's yelling at you, they're over your shoulder and you're looking at this document for the first time. I hope that you won't do that. This is why we're having this conversation. Let's get to this next section here where we talk about recovery environment. If you are certain, and sometimes you're not, but if you know for a fact that the reason why you do have to recover was that you were under a breach, then you need to have an isolated environment that you're gonna recover to. If you don't do that and you use the same network and you are recovering your backup, the threat actor probably still has an account in there. Threat actors will dwell upwards of 6 months in your environment, doing reconnaissance, waiting for you to go on vacation, to forget something…but they're actively looking at ways to get to you. So, making certain that when you're in a recovery that you are doing so to an isolated environment will limit their ability to get to you again. Most of the time, companies that have malware attacks or ransomware attacks, when they come back online, they get ransomware again because of this part. They don't go into an isolated environment and the threat actors are like, yippee, you put it back on the network. Thank you. I'm gonna get you again. Backups and data integrity. Do you have multiple backups of at least two of your domain controllers in your environment? And even if you do, how old is the backup that you use? Do you know that a threat actor was only in your environment for a couple of days? Do you know if it was a week? And even if you have those backups, you have to ask yourself, how good is that data now? If I'm going to back up recover from a two-week-old backup in Active Directory that presumes that there's a lot of other changes that happened from two weeks ago that I now have to account for and then probably do over again. So sometimes it gets to a point where the integrity of that data is no longer valid. Your topology. Do you know all of the relationships? Do you know what your FSMO roles are? Do you know which domain controllers had the highest utilization? Things like that are things that are gonna come up. Do you know what the trust relationships were if you have multiple forests? Let's pop in here. Your DNS. Is your DNS integrated in your Active Directory environment? Do you use an external DNS? Is that external DNS going to be up and running before you start trying to do this recovery versus an integrated architecture? Let's pop over here. So, this is really important. I put this in in yellow because as I was mentioning, the first time I had to do this, they said to me, hey, just make sure you have all the DSRM passwords. Dude, I wasn't here when you guys created that. I just got here a year ago. So I had no idea where those passwords were. But that's critical information because you can't really even do a recovery unless you have that. And then we talk about recovery strategy and logistics. Which DCs do you restore? You know, which ones do you re promote? Do you have the install for media packages readily available? Are you gonna have to create that and then ship it over? It's a lot of questions. And if you haven't had to do this and haven't had to work in operations for Active Directory on the back end, some of this just sounds like a lot of noise but trust me, if you ever need to do this because the worst thing, the worst day of your technical life has happened and that you have gotten breached and you now have to figure out how to do all of this, you wanna be as prepared as possible. And I'm gonna tell you, you gotta do the hard thing folks. A lot of times we talk about doing tabletops and we think that that is sufficient. No, you need a process that you can actually go through, you can dig in with it, do it repeatedly. Practice doesn't make perfect but practice does make progress. And so, having the fortitude to do the hard thing when it comes to do developing the right type of recovery plan for your identity—remembering that if Active Directory is not working, most chances nothing else in your environment is working. So the more that you can practice getting these, trying to do recovery is gonna be helpful for you. A few more, and I won't jump too much into this but things like application dependencies, you know…oops, went a little far here. Which of your applications are reliant on which domain controllers. Real quick story, I was in my first disaster recovery exercise for my last company—and I've had done this at many other companies as well— but in the last one, we used a system called Confluence and my job was you know, we use Confluence to track our disaster recovery exercise. Well, my job was to shut down the domain controllers essentially making the identity no longer available. Confluence went down. And I said, hey guys, did you hard code Confluence to these domain controllers? No. We would never do that, says the Confluence application. But it wasn't working. And I said, hey, we're gonna fail this disaster recovery exercise because we can no longer track what's happening. And why is that important? Again, if you heard me mention earlier, you have regulatory compliance, these are things that your insurer is gonna need to know in the middle of a disaster. And if you can't keep track—and I hope that you're not handwriting these things—if you are not able to keep track, what's going to happen is that you are going to end up missing critical information that is gonna be necessary for them. So having a surety that you're knowing which applications are going where is going to be important. But the most important thing is trust. And I didn't talk a lot about this but the data that you are recovering in your identity, can you trust it? Do you know if the threat actor is still in there in your active directory? Do you know which accounts, any back doors that they have in there? If you don't, then you leave yourself open again, as I mentioned, to being attacked again. Right? So let's talk about what your greatest asset is in recovery. And I like the face that you're giving me this, sir. Because you're like, what does this mean? Well, for me, it's people. We talk about people, process, and technology in these security conferences all the time. Right? But to me, the biggest asset is the people. Because we all know…I mean, you're hearing all these discussions about AI and I'm like, that's great. But it can't replace—I don't believe, that it can replace—the people. We are able to make sound decisions based on the technology that we implement and the processes that we build around that technology. So the people element is gonna help you to make good decisions. When we're talking about Active Directory especially, I say that everyone plays a role in securing AD or whatever your identity provider is. Right? You know, sometimes I'll talk to a customer and they're asking about our products but they say, oh, you know, well, we use Google or we use something else. Whatever it is, everyone plays a role in making sure that it's secure. Right? Your infrastructure and security teams at your company, your organization, they should always be in lockstep. Right? A lot of times when I speak to folks, they may have the infrastructure team who thinks, well, I wanna do this and security is like, hey, hey, we can't do that. There needs to be good marriage of those teams to make sure that your organization is considering all the things that are important. And I have found that the more successfully secured environments are organizations where they work together on all important changes to their environments. If the humans are aligned, the processes around securing your environment are typically also aligned. Process. I told you about this earlier. Do the hard thing with testing your Active Directory recovery. It's unlike just backing up a file server and then recovering from that. That 30-step process has all kinds of things in there. You know, recovering, you know, the SYSVOL, recovering the DNS or if it's integrated, cleaning up metadata, moving around FSMO roles—like all of those things are really they're small but they matter. And if you don't remember the steps to do all of those things that I just mentioned and I look at this stuff all the time, it will behoove you to do the hard thing when you're doing your recovery testing. Your exercises should mimic as closely as possible real-world scenarios. You don't wanna do what Trey, I'm sure, has had to do, and when you had that experience—was it from an outage? Was it …Okay. Right. So it's an attack. Right? You don't wanna wait till an attack to figure out that you don't remember how to reclaim FSMO roles to one of your DCs. That's something that you have the time to prepare for and know what is important and how to do those things. And again—thank you. And then rinse and repeat. Right? Don't do just do it once. A lot of times companies tell me, well yeah, well, you know, we did a disaster recovery exercise last year. Or we did it, you know, at the beginning of the year and it's like November. You should be doing these as often as you can that makes sense for your organization. You know what's important to your organization I hope, but really look at doing at least quarterly, I tell folks, if you're able to because you need to practice. Especially now, we don't live in an environment where it's if you will be attacked. It's when you are attacked. How do you recover quickly? That's going to be critical in getting your business back online. So, in your technology toolbox, you should make sure that you have tools that tell you what your Active Directory security posture is. So those are gonna be things where you're gonna be running indicators of exposure. Those indicators of exposure are gonna show you, especially with Active Directory out of the box, there are a lot of misconfigurations—or just a lack of configuration of many objects—and so this will help you to figure out what you need to be plugging up so that you don't get to a point where you are breached because you just didn't configure it properly. And then, you know, you wanna make sure that you don't do what I said has happened to me in those my mailbox that had 30,000 emails. Make sure that you address them and start with the low informational if you have the bandwidth to do it. Obviously, always jump on the critical things—but we forget the lows and the mediums. Right? And then those are a lot of the times the things that threat actors are using to leverage your environment. And then you also wanna have something that could look for indicators of compromise. Have you been breached and just don't know it yet? Is there a threat actor already in your environment? IOAs: Are you currently being attacked? It's one thing to have a threat actor they've leveraged, they've compromised you but they haven't dropped the payload yet. It's another thing that they could be actually attacking you. They could be exfiltrating some of your data presently. Do you have tools? Do you have any visibility into that? So you wanna look sure make certain that you have tools that can help you with that. Again, this is all gonna be part of your process. Right? That even if you have those tools, you still wanna make sure that you're using those tools in a way that can help your organization. And then there must be a better way to recover because like I said, we're no longer at if it happens. When it happens, how do you recover? And I can tell you this—and I'm sorry Trey, I keep picking on you—but would you have preferred if there was an automated way to do that recovery? Absolutely. Absolutely. Right? And part of it is, the Microsoft process, it tells you what to do to get your first domain controller of your forest back online. Then you have to replicate that for everything else. That's a lot, that's a lot of work. And this is why we say that a manual recovery of Active Directory is something that takes from days to weeks to recover from and I don't know any, any organization that is reliant on making money that can just be out of business until they get something like this up and running. So even if it's automated, you still have to ask the question, can you trust your recovered environment? So any tool that you use should be one minute. Should be something that can also allow you to scan that environment to make certain that what you have just now recovered no longer has that threat actor in there. And if it does, it should be able to bubble it up to the top so that you can see that. So my key takeaways are that: Active Directory Security is critical to your business. Manual recovery is very slow. I want you all to do the hard thing in your testing. And this is key to my heart: People still matter. You are the secret sauce. You folks that are here today, you are the secret sauce to your organization's success when it comes to recovery, when it comes to security. Keep doing the wonderful things that you're doing by being at these conferences, talking to your peers, joining organizations. You matter. We all make a difference in securing our environment. Thank you so much. I'm gonna leave you with my LinkedIn so please feel free if you have any questions. I'll be at our table in the main room, we're when you come in on the right side or in the back. Thank you for taking the time to listen to me today. I really appreciate it. As I mentioned, I work for Semperis and we are an AD security company. Please come talk to us, we'd be happy to help you. Thank you.