Video Transcript

>> ANNOUNCER: Please welcome CEO Mandiant, Google Cloud, Kevin Mandia.

 

>> KEVIN MANDIA: Thank you. Thank you. It's good to see 2000 of my closest friends here today. I have got 20 minutes, but I have 40 minutes of material. But I think you are entitled to as much as we can fit in over the next 20 minutes. Looking at the year in review of 2020. So I'm going to start off with by the numbers what we saw doing 1163 intrusion investigations during the year. Then I'm going to talk about the apex attacker and what they did. Then what top defenses are doing against maybe the top 1-2% of offense that's out there and then we will wrap up with the concerns of the CISOs.

 

So with that, let's get moving with the year in review. Already mentioned the number of cases that we responded to, where we are even as I stand here right now, responding to over 100 security breaches. And these are the intrusions that circumvented common safeguards and the intrusions that got to a scale and a scope where people needed Mandiant to come in and figure out what happened and what to do about it. In addition to learning from the front lines in the intrusion investigations, we have a global threat intelligence group of over 350 folks, I think the slide says approximately 342. We snuck a few more in there. That speak 34 languages globally deployed.

 

When we look at the front-line intelligence that we are getting, one of the things that we have learned is we are detecting faster. We have been tracking the stats since actually even before this slide, you know, shows, and if you look back a decade, it took over 200 days for folks to notice that, hey, we have got a problem. So, when you are looking at global median dwell time, we, kind of, use dwell time to define from the moment you're compromised to the moment as a victim organization you actually know about it and you are aware about it. But as we look at this globally, we are down to 16 days, so we have made tremendous progress in getting faster. Now, we can slice and dice this data for dwell time based on investigation type, geography, and internal or external detection. So let's do that.

 

The global median dwell time in all regions for ransomware, actually, went up. And a lot of you would be like, wait a minute, you just saw a slide from IBM that says you got four days from the moment a ransomware actor is in to the moment you got a problem. What we are learning here is this is from the moment somebody breaks in to the moment you know that they broke in and they are deploying ransomware and I think it's the separation of duty that increased the length of time of detection.

 

There was a time in my career where if you got hacked and ransomware is deployed, the stigma was, you're not that good at defense. What did you do wrong? But now we have professional hackers, 1099 employees – no, they are not that. But you get the idea. You have apex actors breaking in and then you have another group buying the access to deploy the ransomware. And that could be part of the reason why we see a delay. Whether right or wrong, this is what we witnessed. Then you can say non-ransomware cases, tremendous improvement for the moment you are breached and someone is trying to do espionage and steal and pilfer information to the moment you are aware about it went down dramatically by 19 days.

 

Then let's look at it by region. And there you have it. You can all read the slide. We got faster in every region except for one. And when I read the forensic reports from the one, I don't have a good story as to why. Just case type, sophistication, could be the progression of maturity of the detection capabilities in those regions but it went up in one region. And then in general, and this is, kind of interesting, you look at detection internal versus external for ransomware, external detection is faster because a lot of times you have a ransomware actor external to you saying you've got a problem. And then internal detection is faster.

 

Getting to detection, something we have tracked on how do organizations know they have been breached. A lot of folks get drilled, and I was talking to a CISO yesterday, where people want to be the first to know that they have had a breach. That has always been in my career rare. I think in the '90s and early 2000s when we responded to breaches it was always internally detected. But when we started Mandiant in 2004 and start tracking this external detection was over 90% of the time for the breaches that we were hired to respond to. It is not abnormal for you to be in an organization, you get a knock on the door from a third party, whether a government entity or third-party provider of security somewhere and say you have a problem. So, we are the same exact place, 63% of the intrusions we responded to were external detection last year and ten years ago. And I think it's always going to be that way.

 

What we are noticing though, with dwell time coming way down, when you get external notification today as compared to ten years ago, you are getting external notification you believe and is actionable and you can do something about it. And I think for most of us that have been in security for the last two decades, you've all gotten that external notification that wasn't a problem. We are doing a lot better than that now. Bottom line, if a third-party security provider gives you a call or gives you an email and tries to work with you on something, we are just getting more fidelity into that.

 

The investigations in ransomware went down. And I don't know if every other vendor is seeing the same thing but from our perspective what we were hired to respond to, less ransomware cases. This may be a misleading stat but I went through our case reports and in the 1100 plus intrusions we responded to, less than 1% or about 1% had an insider involvement of some kind. But realized the bias to man investigations are that people hire us when they believe they have an external intrusion ordinarily. We do get hired sometimes for insider threat, we treat that somewhat differently. But every once in a while, we respond to intrusion and we can see that somebody on the inside is, in fact, working with somebody on the outside.

 

I don't want anyone to walk out of here and think, my gosh we have an intrusion, who did it, and start looking at your employees. Realize that in my experience, well under 1% of the time did I ever respond to an intrusion and it was insider involvement. Multiple groups identified is quite frequent and this matters because when you first have an intrusion and you responding and you are in that fog of war stage you're trying to figure out what's the zone of potential acumens of this intrusion. If it's ransomware, you sort of know, although you want to know what ransomware group did it and how you negotiate with those ransomware groups, how many of them really abide by their word, and how they do business.

 

Same thing in criminal cases, you will get in espionage you want to know how many groups are in, what do they do, how do you remediate. And it's just amazing to me – and people don't really think about this. You are compromised from one group, a lot of times there's more than those folks targeting you and 27% of the time, multiple groups. We have had four or more in different government agencies and organizations when we respond. This, perhaps, is a big change in my career, is so how are folks breaking in? And I can tell you between 1993 and 1998 – and you’re like my God, he's going back that far, yes. Every time I responded to a breach, publicly available exploit, vulnerable service, mostly UNIX platform, some windows by the end of that time frame, right around 1998-99 we saw more Windows based attacks as Windows was getting more popular.

 

From '98 to about 2002 we saw the transition of victim zero going from vulnerability to I would call phishing attacks. 2002 to 2019 victims spear phishing, literally every single year, spear phishing was the number one way folks broke in and that started to change in 2019. And when you look at it today, right now about 32% of the time victim zero when we know victim zero, it's a vulnerability. Not a zero day necessarily but a one day, two day. And that means it changes your defenses. It means patch management is back on the thing. Anti-phishing is do doing its job pretty well and attackers are going for that.

 

When we looked at it by region, Europe was different, though, spear phishing was still number one for the intrusions that we responded to in Europe. And in Asia Pac and the intrusion set that we responded to there, victim zero was, in fact, either prior compromise or stolen credentials. So, from my experience, we often see geo differences in this category but when I looked at the United States where we responded to most of the breaches, 38% of victim zero was an exploit. So that brings me to what were the top three. Probably no surprise this year as to what the top three vulnerabilities that were exploited were in 2022. Hopefully all of you have patched these, identified if you have them and took care of them. And if you haven't, someone else will find it for you.

 

Zero days is an interesting number for me because we started tracking this literally in the '90s. And when you look at the average zero days or attacks with no patches between 1998 and about 2019, it was low teens. Some years it was eight; some years it was 13. Exploits in the wild is not really common. And maybe it wasn't common because, you know, attackers didn't always need them. They could get in and stay in for a long time using valid credentials and lateral movement. In 2019, we saw 32 zero days in a while and we went, “Whoa, that's a huge spike, double the median, even more than that.” What happened in 2019? Probably defenses got a little better, offense, vulnerability research got more mature and better.

 

We had 30 zero days found in the wild in 2020, 81 in 2021. Marquee blowout year for the offense in 2021. And then this year, 2022, 55. We’re – folks, we are in a whole other planet when it comes to zero-day exploitation since 1993. I've never seen anything like this. So, what does this mean? Again, now you got to be thinking, I just said patch the one days, got to do that. But what do you do about the zero days with no patch? So we will talk about that. And then notice that we cannot attribute the use of zero days to groups all the time. There's a time in my career where we could. Virtually every zero day I saw up to about 2015 for the most part, meaning over 90% were in the hands of nations or folks acting on behalf of a modern nation to conduct espionage and predictable activities. That's changed now where we are seeing financial criminals get it. But the upside of zero days in 2022 we never saw the spray and pray zero day like we saw in 2021 in February when Microsoft had a few zero days in exchange.

 

And by the way, my whole career I almost never see spray and pray zero-day attacks, it's rare, hard to have impact on them. So let’s break these down a few other ways as to who is doing them. China led the way in innovation in doing vulnerability research and finding the most zero-day attacks as we could attribute and I share the CVs being used for cyber espionage and we can only attribute 13 out of 55. But realize when – mainly attribution we are darn good at getting that right and we have a high bar for it. I probably could have put more flags from China on this chart.

 

And then we broke it down by vendor and it doesn't really change. It's going to be Microsoft, Apple, and Google; always the top three because they are the most pervasive. But we have another category grew a lot because whenever you have more than the mean average of between 10- 15 zero-days, while the category the usually grows is the other. So let’s take a look into that other on this slide. And this breaks it down, by the way, by type of platform so it won't add the 55 zero-days because we have another category there. But what I found unique in 2022 is that the other category had all this network security devices and it was my opinion when I read through who was finding zero-days on the edge, in your VPNs, in your firewalls; it was China. They led the way in finding zero-days in vulnerabilities that they weaponized 2022.

 

So let's talk about the apex attack. You know, I read through our forensic reports. The last one I got to had a great password so I didn’t even try to read it for a long time. When I cracked it open and read it, it was an attack into a defense industrial base organization somewhere, and what was remarkable about this is the apex attacker in 2022 in regards to cost to conduct the offense, hands down was this account and this attack coming from a group we could attribute to China based on both infrastructure, as well as the compression algorithm that they used in their file transfer. So, we had a great idea, it was them, we are pretty confident in it. But if you look at this attack and right now all of you are squinting at the eight font. The way this attack was detected was about 14 FortiGate firewalls just didn’t reboot one day. And the question became why were they booting at all? Somebody placed code out of bounds of where you should be able to access code on this Fortinet boxes and made them actually crash and when they rebooted there's a check sum mismatch. That's an interesting detection that you may have a problem.

 

So, your security devices are rebooting and getting mismatches in the FortiOS. From there, the attacker then showed zero-day capability with the directory traversal CV that we published – I’ll never get the number right, I think CV41328, directory traversal is an old thing, right? But if you can write code into a portion of a protected device that you shouldn't be able to write code to and maybe append something on a legitimate file and push it in, from the FortiManager box, it's going to execute and you're going to have a back door in place. And that’s exactly what those folks did to 62 firewalled devices.

 

So now you have an attack plain, access to a company and you are on their security plain and there's no EDR on it and you have written code to places we couldn't even do forensics on it because it's on the protected portion of a device. Pretty smart place to be. But then it gets even more interesting. Then they go and get access to the VMware hypervisor and they had a very sophisticated malware framework to go from hypervisor to guest image and put files into the guest images, go from guest image to guest image and guest image to hypervisor. That malware framework, we are still looking at pieces of it. That thing wasn't created in a day. That thing wasn't created in a month. It was created in many months.

 

 

So, we are looking at one case, zero-days were used, infrastructure without EDR coverage was targeted, in my opinion. Logs were deleted, that is something that was new. When I responded, I have done my 10,000 hours on the front lines doing forensics on Chinese cyber espionage, there were many years, over a decade where there was not a lot of op sec of deleting logs. We’re finding as we scraped memory in the FortiNet machines, looking at the Apache logs that they had – the access logs that the attackers are using sed and just grepping out essentially or tearing out the IP addresses and they use 12 different source IP addresses.

 

Bottom line, this was the apex attack in my opinion on offense. Very hard to respond to, very hard to detect, and there will be more victims than many will be aware of because of its efficacy and how surreptitious it was. So what do you do about it? Because that’s what this conference is about, the threats will change all the time. If you're in an industry like defense and industrial based, financial services, you make something that's critical to the health and welfare of 1.7 billion people, you may be targeted by these types of attacks. So, what do you do? Well, don't ever forget the advantage you do have. You should know more about your business, your systems, your topology, your infrastructure than any attacker does. And there is the exception, is you hired an insider, but the reality is, this is an incredible advantage and it's one that if you can base line normal and structure things right, you can detect anomalies faster.

 

So I went to my team, I always mention Tim Crothers, our CISO whenever I speak because I’m always like, “If you're in the crowd, Tim, please give back the work.” But in this, I said, “Hey, what are we doing to make sure we detect – you know everybody has got the blocking and tackling for the one day, patch management and the health and welfare systems very quickly.” But what do you do about the O day and so he said, “Well, don't forget the one day, focus on the fundamentals.” And I want to just – and this slide I don't have the time to get to all of it. But CISA has a great document on the zero-trust model. And everybody defines zero trust in many different ways. But it's great to have a document you can go to, that's free at CISO.gov. They have that cool mountain logo that I stole and it's a journey and if you just kind of benchmark against the maturity model, you're going to do a lot of the fundamentals right.

I get the question all the time, I’ll point this out. Biggest bang for the buck against ransomware, biggest bang for the buck against any impactful attack is multifactor authentication period. And figuring out a way to get it everywhere and know you have it everywhere with some sort of validation is critical. Now for the apex attackers, it's how do you detect them when products don't stop them? That's fine. There's a couple of things on the side I will point out. That we do I think is neat, I think it’s cheap, and it’s something everybody should do. Fancy word for it by our security staff was honey tokens. I literally had to make the call, “Tim, what is that?” To me it's something we have always done. The fake accounts that are baked, that are cheap as heck to thrown in your active directory, that nobody uses, nothing should use, but it’s sitting there and an attacker doesn't necessarily know when they first get in and they will target your identity architecture when they break in. If you have a few dummy accounts and build rules around them that any use of them is bad, that's a great move. That's your free honey pot for your identity architecture being violated or being insecure.

 

So I am going to point that out. Everything else here you can see. But the manual validation of security processes for me, talk about on the other side. With two minutes left. Wanted to point this out, so what do you see in stage 2 of almost every single breach? We got to track this amongst the MITRE kill chain. It's PowerShell, bad PowerShell scripts. And there’s not a whole lot of software you can buy out there that just immediately instantiates, that’s a good script from a good person and that’s a bad script from a bad person. You have to turn on your module logging, you have to study this, you have to get your security operations folks to get visibility into your PowerShell use, your active directory, or if you're not using active directory or LDAP or something else, you have got to watch all identity use, period.

 

So, that's number one on this slide. And to some extent it's number five on this slide. Get great at those. Good luck trying to figure out file deletion because good people and normal people delete it all the time. You can turn object and access tracking on in Microsoft. You will just get a ton of tough stuff to do. But when you look at this, you want to build stage 2 detection. Concerns of the CISO, I don't have the time to go into. But we report on this every year because we get to meet CISOs when they have to transform their security programs pre-breach/post-breach, where you take security pretty serious but I want you to look at it if you're a CISO and say, hey, did Mandiant miss anything or B, did you miss anything that I need to add to my list? So I could have done all 20 minutes on this and in fact, we could do 45 minutes on this slide alone.

 

Pausing for three seconds because I see cameras taking photos. But I'm sure these slides will be online. And then the last slide, kind of, what are secure – the mature CISOs of the world, the boards, the executives are doing these four things at the most mature security programs. First and foremost, they report risk to their boards in a consistent way. It's not every 90 days the CISO is saying something widely different than 90 days before. I just made the assumption mature security programs are briefing their boards every 90 days. A lot of companies don't get to do that. You have to know the risks that – the assets that matter, the risks to those assets, and then you validate can the risks become a reality. In my opinion, as much as possible we do it every 90 days. You have to do a tabletop exercise, make that tabletop exercise the one that you are most worried about, ransomware that shuts down something that matters. Client data that's stolen so that you know who is involved in your response if the bad threat becomes a reality. What PR firms you would work with, who needs to come to the roundtable to help you decide.

 

And lastly you need to participate in some community so that your security operations is learning from other people's security operations. So, that's as fast as I could do it. Thank you so much for all the cybersecurity professionals, what you do every day. Thank you.

 

(Applause)