Video Transcript

>> SPEAKER: Please welcome George Kurtz and Michael Sentonas.

 

>> GEORGE KURTZ: All right. It’s like we never left here years ago. We’re back at it.

 

>> MICHAEL SENTONAS: Good morning.

 

>> GEORGE KURTZ: Good morning, everybody. Hopefully, you had a good week. And guess we’re one of the last acts on Thursday, so we appreciate you showing up.

 

I’m not sure if it’s just me, or everyone else, but every time I meet somebody that I’ve seen on Zoom, I go, man, you’re a lot taller than you look on Zoom. I can’t tell you how many folks I’ve seen that way. I didn’t realize how tall you were.

 

So, we’re back at it with Hacking Exposed. This is – I don’t know – probably almost 20 years of doing this. And the main theme of Hacking Exposed, which really came out of the Hacking Exposed book which I co-authored, was to talk about topical security issues to show actually how they can be used to exploit current technology, and then, of course, countermeasures that people can put in place. So, with that, let’s jump into the presentation. Here I’ve got Mike Sentonas, who is our Chief Technology Officer, and hopefully we’ve got a good show in store.

 

>> MICHAEL SENTONAS: Fingers crossed.

 

>> GEORGE KURTZ: Fingers crossed. Okay. Disclaimer: They make us put that in. Okay.

 

So, we talked about what we want to do. I guess we had two years to plan this out while we were sitting around doing nothing but Zoom, right? And we really thought about what would make a good demo for RSA? What was a topical issue that companies were grappling with on the security side? And we came up with Kubernetes and containers.

 

>> MICHAEL SENTONAS: We did. And what was really interesting about this, we thought it would be a great demo because, obviously, from a container escape, if you escape the container, you get access to the host, you have full control to do anything. So, we were coming up with a lot of ideas in between Zooms, and just came across -- our team of researchers, who are phenomenal, came across a really, really interesting and pretty high-risk vulnerability, which we thought we have to show. So, we pivoted the whole presentation so we can show you that research.

 

>> GEORGE KURTZ: Yeah. So, let's talk a little bit about this.

 

So, cr8escape is a vulnerability that, actually, we found at Crowdstrike doing research for this. John and Manoj at the bottom are recognized for finding this, and it’s a pretty niche escape. How many folks are using Kubernetes somewhere in their production world? Yeah. Probably a few more than we actually think in here. I think Kubernetes, obviously, and its related technology, cloud native, is, obviously, the future of technology, and there's a lot of concern by CIO's, Chief Security Officers, Dev teams, DevSecOps about what would happen if you can actually escape a container?

 

>> MICHAEL SENTONAS: Absolutely. Similarly, you can see CVS-2022-0811 had a score of 8.8, very high. The good thing is when we show you everything here, there is a patch available, so we want to make sure that you are aware of how this works. We’re going to give you a whole bunch of fixes, so you can think about how we approach the pre-fix and pre-patch, and then we’ll kind of take you through the journey of what we found and why we think it's so interesting and why we wanted to talk about it today.

 

>> GEORGE KURTZ: And the vulnerabilities found in CRI-O, which is a runtime engine that really underpins Kubernetes. It allows you to run the containers. So, let's talk a little bit about -- we’ll get into the vulnerability, but maybe we should jump into virtual machines versus containers.

 

I will start with this. I remember, gosh, it was back maybe 1999-2000, I actually walked into a small little company, there was about ten people, and I think it was Palo Alto, and they had this little sliding door. I will never forget it. It almost looked like a converted house. And they said, we want to show you something, and they fired up this virtual machine, and they were running like Windows NT or something on it. I’m like, wow, that's the coolest thing ever. It was a little company called VMWare, and it was literally ten people. They’re in the news now, potentially getting acquired. But I remember. And I saw that, and I said, man, that is the future of technology. And I thought it was the future of security. Why was I so interested? I didn’t really have a great appreciation of how it could change security. I didn’t have a crystal ball. But I was interested because of doing these Hacking Exposed demos. We actually used to lug around like three cases of equipment, like Spark IPXs in these pizza boxes. Literally, we’d have laptops everywhere to try to make all these demos work. And once we got VMWare, we were able to virtualize it and saved a lot of time.

 

So, what is virtualization versus containers? A lot of folks are familiar with virtual machines. You’ve got a hypervisor, which really emulates the hardware, and then you can run an operating system on top of it, and then you can run any applications you want. The beauty of virt – sorry – a hypervisor is incredible isolation. So, you’re really virtualizing the entire machine from the hardware all the way up. If you compare and contrast that with a container, a container is really focused on virtualizing the application space. So, I want to run Engine X as a discrete application, and I can basically just pull down a container and I can run it. And you can see here in the diagram here, you have a container engine which runs containers, and then, ultimately, it runs the application, and the application – the container has everything that you need to actually run that application, all of the libraries, everything you need to actually run it, and really nothing more. And the whole idea with containers is the fact that it makes it super easy and efficient, but, again, at what cost, Mike?

 

>> MICHAEL SENTONAS: Yeah. Iteration speed, obviously, is huge with containers. You can go and get a container from anywhere. There's libraries everywhere. There's thousands of containers that you can go and get all pre-built for you, plug them in, they’re working within a couple of seconds. But the cost is obviously, from a security perspective, if in the example that we're going to show you, you have a shared host exploit, conceptually, you end up in a situation where a row container, a container that's been manipulated, an attacker that gets access gets full access to the underlying host, and then everything. And at that point, they can set up more row containers, they set up persistence, they can get access to passwords, they can inject into runtimes. Worst case, you execute ransomware. And that's why we think this is -- there's some practical things that you can do to address this, but you need to know what to look for. And for two-thirds of the room that didn't put their hands up, we want you to think about, well, okay, there's a lot of things that I can do. There's pros. There's cons. There's trade-offs. And we're going to talk you through a lot of those as we go through this morning.

 

>> GEORGE KURTZ: All right. So, let's talk a little bit about container escapes and what that's all about. So, essentially, when we think about how containers work, it's really about protected namespace. So, in the Linux kernel, you have a specific namespace, you have protected processes and resources, and the whole idea is it's very efficient. You can put all of your information, all your application, all your processing into a particular namespace, and it's supposed to be isolated. What Mike was talking about earlier is supposed to and reality sometimes don't actually -- don't actually meet.

 

>> MICHAEL SENTONAS: Yeah. They certainly don't come always together. The interesting thing is, from a security perspective, it is quite common when we see these issues, people are focused on building security. I mean, I've never met a DevOps person who's trying to publish an app that -- that doesn't care about security. It doesn't work that way. But a lot of the security we see is focused on the container. It's not connecting the container, and it doesn't always extend to the host due to the trust and the isolation boundary, and that's where you kind of run into some of these problems.

 

>> GEORGE KURTZ: Yeah. When we think about escaping the container, the negative here, obviously, is the fact that once you get out of this protected namespace, you're actually in the host operating system. You're in that Linux operating system, as opposed to kind of the virtual machine that we talked about earlier. And once you get on that host machine, again, the big fear, particularly for cloud native shops, and a lot of people as we move to the cloud, is that if you have one container that's compromised, you can knock out your entire Kubernetes cluster, and more importantly, the host machine that actually runs it, and once you're on that control plane, then being able to move laterally across a production environment.

 

>> MICHAEL SENTONAS: Yeah. The one thing I'll just add to that, just to kind of put a bow on that point, is a lot of people focusing on the container. If you're not looking at the underlying host, you could actually see odd behaviors, let's just call them that because they can be many different things, they may go unnoticed, and that's typically where you can get into trouble here.

 

>> GEORGE KURTZ: All right. Let's talk a little bit about CRI-O, which is a container runtime engine. Again, maybe we should spend a little bit of time on just explaining the technology a bit, and there's probably a lot of experts in the audience. But you've got containers, which is sort of a bundling of applications in libraries, you've got pods, and you can have one container that goes into a pod, and then you have a cluster, which is a group of pods put together, and then you've got Kubernetes, which is really the orchestration layer for a bunch of different clusters and pods that are out there.

 

>> MICHAEL SENTONAS: Yeah. Absolutely. A bit of a primer for those that maybe aren't necessarily familiar with CRI-O. As George said, container runtime engine, open-source, community-driven container. The goal here is to replace the Docker service because it is so easy to use. So, OpenShift uses it. Open container initiative, so you can have a range of different containers come in, you can use them very, very quickly. Red Hat, OpenShift, Sushi. Cubic uses it. There’s a whole range, Oracle, Container Engine. This is very, very common.

 

>> GEORGE KURTZ: Yeah. Okay. So, now we talked a little bit about the setup, why it’s important, why people should care about it. Why it is one of them on risk? And why it is actually so challenging, in some senses, to find this because there aren't a lot of technologies out there that are looking for this. And it's a newer area of security and technology and exploitation, and it just doesn't have the same level of scrutiny of like malware detection.

 

>> MICHAEL SENTONAS: Yeah. Absolutely. Let's get into the vulnerability details. Again, credit to our research team that worked on this. Pretty cool research that they found. And, again, the reason why we gravitated towards this example to walk through with everybody is because we wanted to show you something that actually is quite simple, hard to find, but something quite simple that had quite potentially devastating impacts.

 

So, CRI-O version 1.19 had a vulnerability. CRI-O 1.19 also introduced support for sys control, common concept within Linux, allows you or gives you the ability to -- to control or to set kernel options, to set them, in this case, in this specific context, to the pod. All of the sys control options are passed to a utility called pins. You use pins because it's a lot easier to set those system controls. And it's actually that process that modifies the kernel. What's interesting is that the conversion of sys control options into the pins arguments have no validation, and that's obviously, as you will see, a pretty serious situation that doesn't end well as we go through the demo. If we can just jump to that next slide there. Thanks, George.

 

You know, going into a little bit more detail. When you look at the system controls, they're grouped into safe and unsafe system controls. All of the safe, obviously, system controls are enabled by default. The unsafe system controls are disabled by default. But here's the problem. CRI-O validates the Kubernetes sys control names against a set of known safe options, but it doesn't actually check, in this instance, of CRI-O, the actual values, the values of the fields themself. Pretty big, obviously, oversight in that particular example.

 

>> GEORGE KURTZ: So, this is the common input validation error that we talked about.

 

>> MICHAEL SENTONAS: For 20 years.

 

>> GEORGE KURTZ: Twenty years. Yeah.

 

>> MICHAEL SENTONAS: When you talked about all the boxes and the things that you bring out, you kind of aged us really quickly.

 

>> GEORGE KURTZ: I know. I aged myself. Alrighty.

 

>> MICHAEL SENTONAS: So, here’s how effectively it looks like. So, you can see here, we’ve highlighted in blue, that’s the section that hasn’t been validated. Checks the name, so we can see kernel.shm, forced, but we've actually dropped in a value using the plus character kernel core pattern. And that's where, actually, this becomes a problem. So, you can actually add multiple kernel parameters that go unchecked. Now, obviously, for adding kernel parameters, we can do things with nefarious purposes, and we're going to show you that as part of the demo there.

 

Alrighty. So, this makes it a little bit real. The one thing that we want to talk a little bit about is core dumps and how they come into play with this particular issue. So, core dump, obviously, common concept. Everyone knows that. I'm not going to spend too much time with that. Core dump creates a file in a normal situation. When the Linux kernel or program crashes, the core -- kernel core pattern parameter tells the kernel how to create the file, the text file that has all the information about what caused the particular crash.

 

Now, what's really, really simple about this technique that we're going to show you, you can, with CRI-O and pins, effectively tell the core kernel, instead of creating a text file and writing the output to the text file, you can run a pipe or use a pipe, and then actually tell the kernel when there's a core dump, execute a script.

 

>> GEORGE KURTZ: So, what's interesting about this is, at least to me, is it uses a lot of the techniques, again, that we've talked about for the last 20 years, right? Input validation, the ability to replace different directories, and have things execute. And then, obviously, if you can get a privileged process to run, which happens when you have a core dump, you can execute the scripts with -- with the privilege you need to actually hack the machine. So, there's a lot of things that, you know, may seem new in terms of Kubernetes and clusters and everything else, but, you know, new is old, and still the oldies but the goodies work in just a new environment.

 

>> MICHAEL SENTONAS: Yeah, absolutely. And this is pretty simple. We've got a vulnerability that we've come across, not easy to come across, so I don't want to trivialize that. That was not an easy bit of research. But we've got a core dump here. We found a way that you can core dump and you can execute a script. It's fairly trivial to create a core dump. What we could have is a situation where you’re just core dumping regularly and running scripts.

 

>> GEORGE KURTZ: Right.

 

>> MICHAEL SENTONAS: And it potentially goes unnoticed, as we'll see here. So, I guess, George, you've been doing the Hacking Exposed bit for a while. What does an adversary do at this point?

 

>> GEORGE KURTZ: Well, they're licking their chops at this point, you know. And part of what we also wanted to bring into this is the fact that supply chain issues are so topical right now. We’ve seen a lot of hacks that have taken place in the past. There's a lot of concern about the open-source repositories that are out there. You've got like two people creating arguably some of the most important infrastructure within the internet. And, you know, are they secure? Can it be tampered with? Et cetera. So, we took a little bit of a different approach, and we focused on -- on the container itself. But if you can get access to the underlying host, it's pretty much game over, right? That's ultimately what you want it to do is to get that root access. You can infect and inject other processes into the containers. You can launch new malicious containers, get credentials. And again, just about any incident response that CrowdStrike has ever done has seen some level of credential theft and lateral movement, which makes, you know, identity controls even more important.

 

So, and the last one, which we're seeing more and more of, is cloud-based ransomware. You know, the ability to actually just take out a production environment is -- has a lot of impact and tends to make people want to pay a lot of money to get back up and running.

 

>> MICHAEL SENTONAS: Yeah, absolutely. And this is why this has a score of 8.8. But instead of talking through slides, let's go to –

 

>> GEORGE KURTZ: Let's get to the demo.

 

>> MICHAEL SENTONAS: Alrighty. So, I will come over here. I'll kind of talk you through everything that we've -- we've talked about during the demonstration. So, hopefully we can get the presentation of the demo up there on the screen and we'll kick it off in a sec. There we go. Right. So, let's jump into the hack. And I'll kind of step you through all the stages here.

 

So, we've used mini cube here to set up the environment. And mini cube, as I said before, lets you set up the kernel --Kubernetes environment locally, so it makes it a lot easier to demonstrate this. You can see here in the -- in the demonstration version is 1.22.1, which is one of the vulnerable versions. So, what I'm going to do here is I'm going to just pivot very quickly just to show you one of the pods or, as George said, one of the containers. A pod can have a single container. It can have multiple containers in there.

 

So, let's -- let's have a look at the pod configuration here. This is a modified version of a standard image. How does this sort of come about? There's a couple of different ways that this can happen. As I said earlier at the start, you've got hundreds and hundreds of repositories of containers that you can download. If you do a bit of a search, you can find a container to do almost anything pretty straightforward. You know, you could do something super basic, you know, some typosquatting, set up a site, put a couple of row containers in there. But again, a lot of containers are used to run applications that are connected to the cloud, so there's a lot of different ways that you could end up in a situation where you have a container like this, somebody gets access to make one of the changes.

 

Now what's interesting about this –

 

>> GEORGE KURTZ: This goes back to the supply chain issue, right? If somebody can tamper with a public container that's out there. It's a little different than tampering with the source code, but it has the same effect, you're getting a malicious object in your environment that can run with elevated privileges.

 

>> MICHAEL SENTONAS: Yeah, absolutely. So, let's launch the container. So, pretty basic. Next step, using coop control. Let's -- let's run the pod with the modified image. Everything, obviously, now runs in the background. So, straightforward process to get it up and running there. So, what I want to do then is, you know, I can see here that it's running. Again, I'm going to run coop control here. And what I want to actually do now is I want to shell into the box itself, to the container itself, and just run a basic ls command to see what's actually running inside that container. We can see here that we've -- we've dropped a file in here with the malicious script. We've done nothing interesting to obfuscate that script. It doesn't make for a good demo if you can't see it running. But we've got a malicious script just sitting there.

 

Now where this becomes a little bit more interesting, and one of the things that we kind of want to step you through, is we need to understand and know the path to that particular script. But it's not just about knowing the path, as we talked about earlier. What we want to do is we actually want to drop into the host. That's the concept of a container escape. So, knowing the past of that script actually starts with getting the full path from the host into the container, into the location of the script. Very easy to get that command, or to get the location. Just literally dump the information about where the script is located. And if you're not familiar, all of the containers will have a numeric value. You get the numeric value, and the location of the script, and, basically, at this point, we're pretty much ready to go.

 

Now to sort of make it a little bit more interesting, we're going to go to a second container. What you're seeing here is we're just running that particular script. And because we're running it inside that local container, it doesn't do anything, so it just simply exits out fairly straightforward. But let's pivot over to another container. And what I want to show you here in this particular example, we're doing the same thing, let's have a look at that particular pod and look at how it's -- how it's set up here. You should notice a couple of things that we talked about earlier, and I'll just highlight it here for everybody. This is where the issue comes into play. We've got the name here, sys controls. We've got the name here perfectly fine. It's validated. It's a safe command. But the value itself, you'll notice just underneath, we've dropped in the plus kernel core pattern and the full path to that malicious script, and that goes entirely unvalidated. That parameter will be passed, and it will be run with absolutely zero validation. I keep saying that, because when you kind of look at it enough times, I'm sure people are going, why, like that just seems so silly. Yeah, absolutely, it does seem so silly. But that's actually the problem that we've got here in the particular environment. That's the floor that we wanted to demonstrate to everybody today.

 

So, kind of moving along here. The thing that we want to do at this point is to start to really build out the demonstration and show you how this actually works in process. So, very easy at this point to just launch the pod, launch the container and run it, which we'll do right now. So, using the coop control create command, we’ll kick it off. And what you'll see here is nothing really happens, because in the demo here that we've got, it's not designed to do too much. But we can see here that mistaking containers running in the background, we're creating the new container, and it's -- it's also running. So, if we have a look here, and we do the same thing, we shell in here, what we can see is that we've set up the container to effectively process core dumps. So, I just wanted to show you that -- that command there. That's what obviously allows you to do the core dumps.

 

Moving along here on the demonstration, what we want to see as we run this out is we want to set up a situation here where we're going to actually cause the core dump. Very, very trivial thing to do. So, the one thing that I just wanted to talk about is to do it, all we need to do is effectively a basic segmentation violation. If you are connected to the cloud, if this was a container with cloud access, again, very, very trivial, but I'm going to run it obviously on the local host machine for the demo here. We do that segmentation violation. We've triggered a core dump. And what's happening in the background is, remember, we're no longer executing a text file with the core dump information. We've executed the script. And what you can see here from Netcat is we've got a connection back to the attacker. And we can see here we're running as root. We can see here that the host name is mini cube. So, we've dropped into the underlying host. And at this point, it's game over. So, what we can see here is all of the containers that are actually running. And George took you through that before. If we can switch back to the slides there. At this point, we could drop all the containers. We can inject into those containers. We could execute or start another row container. And if you're thinking that looks really simple, it's really simple. There was nothing about that that was too hard, other than the fact that we found that vulnerability without the validation. And for everybody running containers, this is a very real scenario that played out, like I said, a vulnerability of 8.8.

 

>> GEORGE KURTZ: So, how do we detect this, Mike?

 

>> MICHAEL SENTONAS: Well, there's a number of things that we can do to detect it. Obviously, we want to show you, as part of the demonstration, how to do that, and I'm going to show you a demo of that. But there's a whole bunch of things that you can do. There's a whole bunch of detections that you can run. I'm just setting up my machine here for the next demo. There's a whole bunch of things that you can do manually, which we're going to talk everybody through. And there's better -- there's things that you can do along the way to make sure that your environment is safe and secure. So, let's go through a couple of those response detections.

The first thing that I'd say, and I think this is hopefully basic for everybody, monitor for all unexpected activities, whether it is a traditional endpoint. Whether it's a virtualized machine, whether it's a container, you always need to be – it’s the concept of Vidya. It's looking for unexpected activity. Programs that you don't expect to run, container images that come from registries that you don't know, entirely new images, weird configuration options, they should be things that you're programmatically looking for.

 

>> GEORGE KURTZ: And I think this is one of the areas where when we talk to customers, they are, they should be incredibly concerned about their security around their pods in a development environment, because if you can get a real container into the into the whole DevOps process, that's a bad thing. You don't even have to run it. You can kind of infect it, and then wait for someone to actually put it in production. And by the way, it's going to be copied and replicated. That's the beauty of cloud native applications. You can just change it at the source, and you can replicate it any number of times across your entire environment.

 

>> MICHAEL SENTONAS: Yeah, absolutely. Some of the other suggestions here, as we work through, you kind of see where we're going with this. Some of it's easy for me to say. Practicality is not really there. And what I mean by that is doing things like using or eliminating the use of sys controls. If it isn't needed, or it's not expected in your environment, don't run it. Again, as I said, easy to say. It's very, very powerful and very, very much user needed utility.

 

>> GEORGE KURTZ: And often what we find, our dev teams just kind of scramble to write their own tools and try to figure out what changed and what hasn't changed. I think it's trying to understand automated ways to do that.

 

>> MICHAEL SENTONAS: Yeah, absolutely. And for the people that said they're running Kubernetes, you know, are you doing things like performing regular scans to see if people are using sys controls in every container? If so, why? Are people using the pins utility? How are they using it? What are the commands that are being passed? Are you logging and keeping track of those commands that are -- that are absolutely -- that are running in real time? You know, let's go to the next slide there.

 

Let's start to think about preventions, patch, patch, patch. You know, obviously, we can't say that enough. You know, every time you have patch panic Tuesday, I appreciate everybody's rushing to work out what to patch, but something like this is obviously very serious. What do you do if there's no patch? What do you do if you can't get a patch out in time? Well, obviously, you need to do things like minimize the attack surface to limit anyone trying to expose that. So, we've got a whole blog on this. We're not going to spend too much time on this. We've got a blog with information about what the -- the patched versions are. But, you know, this is a really super important point that I said before. You can't focus just on the container. You have to focus on the underlying infrastructure as well, and everything that you do needs to combine that entire environment.

 

And then the other one here, you know, it's not a term that I normally say, and not a term that I enjoy saying, but you’ve got to shift left. This is the whole concept of having a robust container, software, or SDLC. Doing things like image scanning, doing things like pod spec, spec scanning, implementing image signing to make sure that you don't have rogue images, making sure that users can't publish images directly to the production registry, this is basic blocking and tackling. It's as simple as that.

 

>> GEORGE KURTZ: There's a lot of process that goes around it. And again, I think it comes back to, you know, we've been doing this a long time, and RSA has been around a long time. It comes back to the people, processes, and technology. You know, the best process in the world -- best technology in the world, if you don't have the right process to be able to create these pods and turn things that are immutable and understand if there is an exposure and be able to capture it and then act on it, you're going to have a problem.

 

>> MICHAEL SENTONAS: Yeah. And this is just a basic start. This is the absolute bare minimums.

 

>> GEORGE KURTZ Okay.

 

>> MICHAEL SENTONAS: So, one more. There are a couple of things that you can do. You can do things like create a wrapper for the pins utility to make sure that you do things like strip the additional parameters. There's pros and cons here. You can strip the parameters, but those parameters are legitimate. They're safe parameters. So, when you strip the parameter, you strip everything. Not that elegant. The other thing that you can do is you can apply a pod security policy to block system controls if they're not needed, and I'll show you a couple of those examples. So, let's cut back to the demo.

 

>> GEORGE KURTZ: All right. Let's talk about the fix.

>> MICHAEL SENTONAS: All right. One more to go, and then we'll -- we'll start to wrap up. So, let's -- let's have a look at the demo here. Let's have a look at the fix. One of the first things that we want to do is we want to look for the presence of unexpected or malicious container. So, easily do that. You can see here the command, so run coop control, get pods, dump the JSON information, and you'll get a list of all of the information. You can see what pods, what containers are running, really simple to use. And you can use this, obviously, to build an allow list, if you will, for reporting. So, basic thing that you should be doing to see what containers are running inside your environment. Again, we're going to use get pods here.

 

Similar method to detect unexpected sys controls. And in this particular example here, we want to see what the -- what's running in -- in each running pod. And as you can see here on the screen, you can -- this allows you to do things like, again, alert controls, or set up alert creation. If we see something new, send an alert. If something's running that doesn't fall into an acceptable list, again, set up an alert. You can also automate response as well and -- and drop them. So, let's -- let's go a little bit deeper here.

 

And what we want to do is we want to see a little bit more about what's running. And you can see here, using these commands and starting to pass what -- what comes back. We can see here the value here of the plus kernel core pattern. And this is the example that I showed you earlier. So, we want to start to think about how we actually do some basic fixes for this and start to leverage the -- the pins wrapper and do a couple of interesting things here.

 

So, let's -- let's SSH onto the machine here. What we're going to do in this particular example is I want to look for the -- the pins wrappers. So, you've got pins wrappers that come with -- with mini cube, and we can use those wrappers to effectively drop the parameters or start to create a bit of a safer environment. I'll kind of walk you through what this looks like. So, let's have a look at the basic version. This is the one that comes available to everybody here. So, the basic version here, a couple of things, those of you with a bit of an eagle eye, have a look in the first third of the screen, you'll see echo skipping, the S value. We're basically dropping the parameter itself. Not the most elegant thing to do, because if we actually go through the process and demonstrate how this wrapper would actually work, what you'll see here, modified arguments will be dropped.

 

So, let's execute and run the container. And what you'll see here is the -- the set -- the hyphen s or dash s. I went too quick. It is dropped and the parameter is no longer passed through. But what you'll see here is when we want to actually run legitimate commands, the legitimate commands are also dropped. So, really, really horrible way to do it because you effectively can hamstring yourself very, very easily. So, what we want to do is we actually want to look at the more improved version. And the more improved version, instead of dropping everything, does things like dropping the plus, dropping the equals character. So, when we run things using that pins wrapper, we're actually dropping just the arguments that are going to cause us the problems. So, if we run this demo again, and we kick this off, you can see there, the dash S is there. We can see here, the additional parameters. The ones that we don't need a dash s, I should say, is still run. It's still passed through. In the previous demo, if you missed the quick demo, dash S was dropped because we're dropping everything very, very horribly. So, in this particular example, the correct argument is passed through, and everything runs perfectly fine. So, it’s kind of the way that you want to run this particular demo.

 

The next thing that I want to do is, again, get a little bit more elegant with -- with the way that we do things. So, you know, what we want to do here is tell CRI-O to use the pins wrapper but do it via the CRI-O.com file. And in that particular example there, it's done with the pins path option, and you can see that here on the on the screen. We'll go ahead and do this with our improved version and restart CRI-O to take effect. So, what we're doing is we're just making sure that anyone that uses the pins command is using the new improved wrapper that we just walked you through. So, to do that, we need to effectively restart CRI-O. And when we restart CRI-O, we'll show you that demo again, and we'll kick off -- we'll start the container. And we'll run the system controls.

 

So, you see there we’ll restart. What we're also going to do, just to make it really simple, is we'll output the -- the --what happens when we run pins. We’ll output to a text file, so you can actually see that run in real time. So, let's -- let's kick off the previous environments that we showed you in the hack demo. So, we'll kick that off again with -- with coop control. So, we'll use coop control, create. We’ll kick off the container with the sys controls, and we'll dump the output there. And what you can see there is the safe commands were created and they were run. And you can see here, we've passed the correct parameters, and everything has run perfectly fine. If we run it again, and we run it with the hacked container, well, obviously, we would expect nothing will happen because we've stripped out the arguments. And you can see here that nothing's going to run because we've effectively removed all of the components that would cause the problems.

 

The last thing to do here is just to show you quickly, there is a pod security policy block option that you can configure, and you can start to add some additional controls and safeguards to limit what people can do and create a safe environment for yourselves. So, pretty straightforward. And again, as I said, we've got a blog, and we've got a full write-up. So, if you want to work out how to use these wrappers and create your own, that is available to you all. So, let's jump back to the slides.

 

>> GEORGE KURTZ: Yeah, I think that's important. We do have a blog that goes into all this detail. You know, we kind of ripped through it on the screen. But probably the takeaway is, you know, if you're a security practitioner and you're talking to your Dev team, or you're auditing this environment, is to really understand some of the exposures and pins, make sure they're using the right configuration. Obviously, a lot of logging and monitoring that we talked about.

 

>> MICHAEL SENTONAS: So, yeah, one quick thing to jump in and say, for those of you that aren't necessarily comfortable with all the commands, we wanted to show you the raw commands. We could have done everything by showing you a product, which, obviously, in this -- in this concept of presentation, we didn't want to do. But with an appropriate container security product, with a cloud workload protection product, with cloud security posture management, everything that we showed you would have thrown detections at every stage. Certainly, without product, I’ll say, we'll throw detections at every stage. So, if somebody is trying to run sys controls, that's logged, and there's a detection. If somebody's trying to use the pins technology or tool, I should say, and they're trying to pass commands that would cause a problem, then you have obviously a technology that can do the response. So, everything that we show you can be easily managed inside a corporate environment.

 

>> GEORGE KURTZ: I think that's an important point, and one of the things that we try to do at -- with Hacking Exposed here is take what we see in the real world, and we've seen a lot of these techniques in the real world, you know, in practice and put it into, you know, a quick 30-minute, 40-minute type demo here.

 

So, let's summarize. We're just about done here. So, containerized workloads and orchestration systems like Kubernetes, certainly here to stay at it. For me, it's the future. Obviously, we continue to build on that from a technology perspective. And as I've -- I've always said in the past, security is really complex because it really parallels the slope of the technology curve. So, I remember in one web 1.0, right, we're in 3.0 now, I think was just like, hey, a basic website, you know, not much to it, simple firewall, packet filtering, routers, those sorts of things, then we got web 2.0, Read Write, interactive, and now we’re at 3.0 with sort of crypto thrown in.

 

So, at the end of the day, as a community, we really just need to focus on defense in-depth. The proactive controls, which we talked about, being able to prevent that supply chain attack, that third-party software from running. When you think about DevOps, and I don't know, we probably have a whole bunch of releases on a daily basis, just crazy how fast it goes, you've got to be able to make sure that you got fast patching, you're updating your containers, and you're running a robust runtime configuration, monitoring technology, and being able to automatically remediate this in almost near real-time because you’ve got to keep everything up and running from a production environment.

 

So, hopefully, we've given you -- I know we went through a lot here. Everyone may not be a Kubernetes expert. But really, in summary, we wanted to just demonstrate what you need to be concerned about. And the same way that, in the past, we've actually demonstrated VM escapes, we've demonstrated now container escapes, and it's certainly something that should be on the risk radar.

 

So, with that, we'll go to the contact slide. You can grab us on LinkedIn. Obviously, the CrowdStrike website and blog has a lot of information about this. And, again, for me, personally, I'm so excited that I was able to come to RSA this year after a couple of years of Zoomarama. And, Mike, any parting last words?

 

>> MICHAEL SENTONAS: No. Hopefully, everyone saw some of those recommendations. And George made a couple of comments earlier that hopefully everyone caught. All the advice is the same thing that everybody does every day of the week. It's looking to prevent unexpected applications. It's looking to understand what's in your environment. It's looking to patch as soon as you possibly can. All of the things that I like to bucket down as just good hygiene. There’s no difference in a container world. And that's really what the takeaway was that we wanted to sort of impart on everybody. A lot of serious impacts of this particular vulnerability can be easily mitigated from a technology perspective with the right tools. This would just never happen in your environment. So, that's the big takeaway for me. Great to see everybody.

 

>> GEORGE KURTZ: Right.

 

>> MICHAEL SENTONAS: And enjoy the rest of the conference.

 

>> GEORGE KURTZ: Yeah. And I know they’ve got the forms you’ve got to fill out, the eval, so we appreciate those. And thank you so much for this, and we look forward to seeing you next year.