Trees Falling in a Forest: Chronicles of a Data Breach
“If a tree falls in a forest and no one is around to hear it, does it make a sound?”
Although the sub-title for this article is a philosophical question, asked by George Berkeley in 1710, it proved to be completely relevant over 300 years later, while I was working with Azure Security Center.
A little background…
A Microsoft study shows that attackers take, on average, just 24-48 hours to penetrate an environment after infecting the first machine. In addition, attackers can stay in the penetrated environment – without being noticed – for up to 99 days on average, according to a report by FireEye/Mandiant.
These sobering statistics offer a gloomy landscape for organizations who need to adhere to strict regulations like PCI DSS and GDPR. Yes, I just used that buzzword. Now, this blogpost is not about GDPR fearmongering. That would be too easy…. Instead, let’s look at how one company coped with an actual breach.
The environment I was working in can best be described as a hosted environment of low IT maturity. The business is to allow customers to use a Windows desktop application, thus offering remote desktop (RDP) connectivity. Because the application needs administrative privileges to be run and each customer needed three servers, each customer was provided their own infrastructure and one of the servers was promoted to a Domain Controller. In Active Directory terms, they planted a forest of trees.
To allow connectivity on the road, too, RDP was enabled without restrictions. To save cost, admins had previously migrated these hardware server instances to Microsoft Azure using Azure Site Recovery.
I know, right.
Their compliance officer wasn’t too happy with the situation, either.
That’s where I came in. As I wasn’t looking forward to battling hordes of stubborn admins, I decided to go the compliance route. There was this new tool I could use to pinpoint exactly what’s wrong on these Azure-hosted machines and allow it to be input for a Governance and Regulatory Compliance (GRC) process.
Azure Security Center
Azure Security Center provides unified security management and advanced threat protection across hybrid cloud workloads. With Security Center, you can apply security policies across your workloads, limit your exposure to threats, and detect and respond to attacks.
As part of the GRC process, one of the admins clicked on the New alerts & incidents tile in the Azure Security Center portal, a couple of times per week. The admin would also report on the actionable Recommendations each month, especially how they progressed to limit the amount of recommendations (but not by dismissing them). Common recommendations across their 100+ VMs were to apply updates and to reboot after updates.
As part of one of the monthly GRC meetings in the above process, the admin shared my ‘Assume breach’ mentality:
The question is not ‘if’ these machines are going to be hacked, but rather ‘when’.
A couple of days later I got a phone call. Azure Security Center reported an incident. One of their hosts was hacked. As we investigated, we found the following vulnerabilities the attacker leveraged:
- The default network security group rule to allow RDP to allow traffic from all hosts on the Internet was not altered to restrict traffic only from known IP addresses.
- One of the end-users had a weak password, that had been in use for over two years without the person ever being prompted to change it.
- The host was missing a couple of updates.
After creating the initial bridgehead, the attacker had uploaded several files, but had not yet created the processes to run them. We deleted the files and reset all passwords to buy time to adequately address the situation.
The next day, we got a call from Microsoft. Not the fake ones we usually get called by, but the real Microsoft, this time… Because this organization was a Premier Support customer and the contact info on the subscriptions was current, they were able to contact the organization regarding the incident, that they, too, had deemed actionable.
After the initial damage control actions and communications, the dust settled. Needless to say, we architected a completely new solution. The breached customer was the first one to use it.
Without Azure Security Center, we would’ve never detected the breach. In that case, the Active Directory tree in the implementation for this one customer would’ve fallen without us hearing it. Because all hosts were in interconnected virtual networks, in no time the entire forest might have chopped down.
The ironic part is that the recommendations we could’ve followed to protect the original setup were all part of the free Azure Security Center offering.
What’s keeping you?