Video Transcript

>> ANNOUNCER: Please welcome Tanya Janca. 
  

(Applause.)

 

>> TANYA JANCA:  Hi, everyone.  This is the same disclaimer that you have probably seen many times before.  And so, just so you know, I guess they're trying to say don't try this at home or blame us if you do try it and it does not work out. 

 

I’m Tanya Janca and I'm here today to talk about DevSecOps, and to talk about things that I have seen gone wrong over and over and over again.  I do research with IANS research. And I do calls with different companies. And over the years I’ve worked with, I looked it up, over 300 different large enterprises.  And so, I'm going to go over the 15 things I have seen go wrong just over and over again, how you can spot them, and how you can make sure, most importantly, that they don't happen to you. 
   

So, now we've done a disclaimer so we're going to talk about DevSecOps.  Tried, tested, and failed approaches.  I find we often talk about best practices and it is like, that's cool we’re supposed to do that. But why?  So, we're going to talk about why, yes, exactly.  So, let's go.  Let's go. 
   

Good morning, RSA.  How are all of you? 
  

(Applause.)

  

>> TANYA JANCA:  Yes.  Thank you.  Thank you.  Awesome. 

 

Okay. So, you are probably like, “Who is this lady?”  I'm me. I’m Tanya Janca.  I'm the head nerd at We Hack Purple, also known as CEO and founder if you feel like getting really official.  And I do security training, and AppSec training, and security champions as a service. I wrote a book about AppSec.  I have worked in IT a really long time.  In July it is 27 years.  I did a bunch of stuff and a bunch of things.  And the summary is that I'm a big nerd on the internet.  I'm really excited about the security of software. And I really want you to get more excited about it. 
   

So I feel like, I feel like the IT security industry is awesome at network.  We are amazing at locking down desktops and all that kind of stuff.  But when it comes to software, we are still kind of working on it. And I feel that over the past three years, we’ve gotten way better. But there is still a lot of work to do.  And so, that's what I'm kind of obsessed with right now. 

 

And with that, I want to show you resting AppSec face.  Like, “You did what now with my app?  What have you done?” So, whenever I go work with companies I often find they'll do things where I'm just like, “Oh, no.”  And so I was like, “What if I write -- instead of just keep writing about what you should do and what's really good,” and all of that, I was like, “What if I start talking about what was going wrong and how you can avoid it.” 

 

And so, my definition of DevSecOps might not be the exact same as yours.  So, I like to say it’s an AppSec person who works in a DevOps shop.  So that means to me I still need to threat model. I still need to, you know, give them a bunch of security requirements. “Oh, cool you are doing serverless? Awesome. I need these two things to be part of your project.  And FYI, I'm going to do this, this, and this with your serverless app to make sure it’s safe.” Right? 

 

But a lot of companies their definition is, that’s more common, is it’s the AppSec person who owns the tooling in the CI/CD.  So, how many people here have had a vendor tell you, “Oh, just put my tool in the CI/CD. It will be perfect and all your apps will be secure, it will be so awesome and easy.”?  Any hands? Okay, I'm seeing hands. Awesome. Yeah.

 
    Of those people, how many times was it perfect the first time?  What, no hands? Come on. If you’re at home, just FYI, zero hands. Okay. That's fine.  That's fine. 

 

So, I want to talk about how DevSecOps isn't easy.  Sometimes it goes wrong.  And I want to talk about why.  So, okay, so a thing in the security industry that I have seen a problem with, we're awesome at keeping secrets.  And we stink at marketing. 
   

We have marketing people that are good at that, but we security folks really stink at telling people what's important, why it is important.  I have met a lot and a lot of people who are 20 years older than me, or older, that are never doing updates.  Because they are like, “Oh, I was told never to click on pop‑ups, you won't trick me.” And then they haven't updated their phone in like five years. 

 

I'm not kidding.  Like, I’m really, really not kidding. And so we’re not giving the good reason of why. And because we are so good at keeping secrets, a lot of us never share our lessons learned.  And this is a big, big, big problem. 

 

So, companies don't want to look bad.  They don't want you to know they have vulnerabilities.  FYI, you all have some. Secret’s out. There is no one that has zero, like zero vulnerabilities in prod. There’s no one like that.  Unless they are not testing at all so they think they have none. 

 

And so, I want to share why. And because I worked with so many companies it’s not clear which ones did what, or whatever. But I think it is really, really important as an industry. And I'm hoping my example of doing this is that more people will share what went wrong. 

 

I used to work in the Canadian government and someone asked me to do a podcast about things that went wrong.  And I’m like, I have one employer for 13 years.  It will be really obvious who I am talking about.  So, I'm hoping as an industry, at some point, we can find a way we share more so we don't fall for the same tricks over and over and over. 

 

But I digress. Let's get to the reasons. Okay so, number 1, breaking builds on false positives. Or some of you, depending upon how old you are, the boy who cried wolf. 
   

I have so many organizations first of all they are like, “I can't wait to break all the builds and stop those damn devs.”  And they act like they are a parent and the developers are two years old and that they are like in the cookie jar. Or something like that. Opposed to a whole bunch of professionals that are doing their jobs the best they can. Who have lots of pressures to get things done in a certain amount of time.   

 

And so, sometimes I see them putting tools in. And so, I have fallen victim to this.  I have bought tools where the promise was zero false positives.  You won't need to do any tuning.  It is going to be so fast. And I remember buying a product where generally it takes like six, eight hours to run this product and it still took two hours. And they’re like, “It is lightning fast!” And I’m like, “For a developer, anything longer than 15 minutes and you are archaically slow. That can’t go in my CI/CD, I’m so sorry.”

 

And so, if we're breaking builds on false positives what we are actually doing is breaking trust. Like every single time we are breaking trust.

 

And I find that a software ‑‑ so I did this. I know. I’m part, or I was part of this open-source project called OWASP DevSecOps, so like sloppy DevOps. And I was like, “I'm so cool.  I'm putting this dynamic scanner in. I’m totally awesome, it’s going to be amazing.” And I’d run a bunch of scans and it was cool. And then my friend who was on the project, Abel, he was like doing some coding, as we do, and he submitted it and it broke. 

 

It broke and said it had SQL injection and it didn’t. And he didn't say anything because we are in different time zones so it was super late for him. So, in the morning I was giving a demo and it broke. And it broke on a false positive. And I looked so stupid. Because I was like, “Look it is working. It is catching.” And then I looked and I was like, “Oh, no. It is not even real.” 
   

And so that was the first time I learned that lesson.  And a lot of you have probably learned that lesson. And so, it is very, very important we test our tools a lot before we turn on breaking.  That's how you avoid this.  You start with testing, not even in the release pipeline, just in a copy of the pipeline.  You run a whole bunch of tests there.  Once it is going really well, then you put it in the actual release pipeline, the one that everyone else is playing in.  Then after months of that and it is totally fine and going fast and everything is good, then you might turn on breaking. 
 

So that's No. 1.  Let's look at No. 2. 

 

Okay.  Turning tools on without testing them.  So, this has the same answer as the last one.  But I see a lot of companies they'll buy a product and the product costs like a hundred thousand dollars. 

 

I once worked with a bank and they had bought a product and they were paying 2 million dollars per year for this product.  And they are a very, very big bank in Canada.  And we only have five, because we are little and tiny and quaint.  But, basically, they turned it all on immediately. Because they spent so much money on it they were like, “We want to get return on investment immediately.”  And so they didn't do all the testing they should. And they didn't look at all the things they should. 

And she was like, “And once everyone stopped yelling at me…” 

 

So, one of the things that we can do is practice outside of the pipeline.  Honestly, when I buy a tool, I pick the dev team that likes me the best and I start working with them.  And I just scan it manually, like totally outside the pipeline. I test it like that.  Then I am like, “What if I clone your pipeline?  Can I try with that.” Then eventually I am like, “What if this went in your release pipeline?” Then if that’s working and everyone is getting along, then I show all the other teams. 

 

And I know you are like, “Oh, that must take forever. That must take weeks.”  It does. But you are allowed to buy like a graduated licensing.  I know all the vendors right now are upset. But you can say like, “Oh, we want to buy like five licenses for the first six months. And then, assuming this goes well, then we’ll sign for the 100, 200, 300.” Or whatever. Like a proof of concept is not necessarily enough.  And buying like a small handful of licenses and trying it out, making sure it’s cool, it might save you a lot of money, and time, and angry developer conversations. I try to have nice conversations with them, and I hope all of you do, too. And so, I try really, really hard.  We're going to come back to this a few times where it is like, “It’s like Tanya was a developer for way longer than she did security or something.” 

    Okay. So, number 3, artificial gates.  Has anyone – okay, so when I was a software developer, the security team, I refer to them as The Department of No. Because that's all they ever said. They weren’t, “No, not this.  But you can do that.”  It was just flat, “No.”  And I would say, “What am I supposed to do?”  And they would say, “Well, you should know.”  Which meant, “I don't know and I feel insecure about it.”  It took me a while to learn that.

 

And so, when DevSecOps came out (inaudible 0:10:44) a lot of them are like, “I can make some gates. I can finally make a gate. I’ve wanted to have a gate and I wasn't allowed for a while, and now I’ll make a pretend gate.” And basically,  like I literally, I have heard people that I respect and really like and they are like, “I just need to slow them down.”  And I’m like, “That's why they will go around you.” 

 

Developers – so, when developers obey security policies it’s because they are politely being obedient.  None of us have to. None of us have to.  We can go around pretty much all of your tools whenever we want. Most of them.  Even the fancy ones, a lot of them.  I still see people all the time, and I will be with them and I will be like, “Oh, I guess we have this blocker.” And they are like, “No, no.  I'll just use my workaround.”

And we all call that an exploit.  And so when you are like throwing up all these fake artificial gates, that are not part of policy, that no one else agreed to. And you are just like stopping things to slow them down because you can't keep up. You need to have a conversation instead of making fake gates. Because guess what, they’ll turn them off because they can. They have the power to turn all of that off. 

 

Okay, so number – yes. Instead, create a policy, actually speak to people.  Don't abuse your CI/CD pipeline, please. 

 

Okay, so next, missing test results.  So, this is a weird one. 
And so, when I first started doing security, I remember running a scan and I was like, “Great, I have the results.  I'm going to e‑mail this to the devs.” And they are like, “Whoa, you can't share that with them.  How will they know how to fix it?” They’re like,  “That's too much information.  That's sensitive information.  We can't share it.  We have to have meetings and talk about it and send a bunch of e‑mails.” And I was like, “Can't I just like make tickets out of this and put it in Jira so they fix it? That's where they live, they live in Jira and VS Code. That’s where my devs live.” They’re like, “No, no, no, no.  We will pick like one from each team and we will give them an account. And they can RDP, like remote desktop into this machine. Then they can log in and then they can do this.” 

 

No, none of them are going to do that.  We don't have time for you.  I need the test results to show up in my CI/CD pipeline. 
I need my test results in a place that the developers like to be.  And there is Bugzilla, there’s lots of Jira equivalents. Whatever bug tracker you are using, I want my bugs in there.

 

Whichever IDE they are using, Visual Studio Code, Eclipse, Intelli J, it doesn’t matter, I want to be able to show them stuff there.  I’d like them, ideally, to be able to run the tool themselves there.  And so, if they can't see the test results and they have to attend a meeting and read a bunch of e-mails and then they get a PDF that's like 200 pages long, these results don't count. 

 

I don't care if your tool can find things very quickly, it is very accurate, if it is locked away on some other very different place that they never look. I need to be able to move it easily and by that I mean automate it, going into wherever they're sharing their bugs.  Or I'm buying another tool. I’m sorry, it’s not happening. 

 

I the AppSec person if I'm serving 2, 3, 4, sometimes 500 devs, I don't have time to go get everyone's results and e‑mail them to them.  Do I look like an administrative assistant?  Don't answer that. Like I have things to do. I have important work to do, right? Like, I'm going to go do a threat model, instead, with that hour.  It’s ridiculous.

 

Anyways, so this is a pet peeve of mine.  And it turns out that most developers don't love this.  And as I have talked to more AppSec people, they don't like it either.  So if you have a product that happens to give results and you lock them away, think about it. 

 

Okay, so next, run away tests. Okay, so I also did this by accident. And I wasn’t supposed to. So, again with DevSlop, the DevSlop team, if you are listening, you are so patient with me. They’re like building code and doing stuff and I’m like, “I'm going to add nine tools to the CI/CD.  I don't understand how to tune any of them.” 

 

And so I remember, like I was creating this demo and I was going to Israel and I'm really excited.  I'm going to speak at like this security conference in Israel with all the awesome hacker people. And I put in this tool and it would just run like four or five hours.  And I remember my team was like, “So, we have 30 minutes on stage.  What are you going to do?”  We ended up just like filming and doing that timeline lapse and making a silly montage video of us like drinking coffee and being ridiculous for the three hours that passed. 

 

But this happens a lot. And so, it is really, really important that we test, again like I said before, like probably since this is number 5, I have probably said it four times.  Test outside.  But if a developer tells you like, “Listen, it is not ending, can you do something?”  Listen.  Go talk to them. 
   

So, I have seen tests run 18 hours where the last build it only took three hours.  But apparently, enough code has changed that the tool has decided it is going to do all the code, not just the part I changed, et cetera, et cetera. 

 

And so, I try to be approachable to the devs.  I have learned not to yell.  When they tell me stuff, don't go, “Oh, my gosh. You should not have done that!”  And this is hard.  I'm very bad at poker.  I always just like look at my cards and I’m like. 

 

And so, I have learned to be like calm and approachable and try to get people to come to me. And if someone comes to you and they are like, “Listen, I don't know what happened yesterday, but it was running forever, so we just turned it off.”  Instead of yelling at them, which I’ve seen people do, say, “Okay. Let me come down and tune that for you, I'm going to fix it.”

 

And sometimes that means yanking it out of the pipeline for a while, while you figure out what's going on.  Sometimes you have to call the vendor and helpdesk gets involved, et cetera. But if a test runs a really long time, that derails everyone's work on that team all day. 
   

Like I remember, I was doing something and then Abel, on my open source project team, he’s like, “Hey, so I would like to kick off a build but you are fooling around doing something. Hello. It’s been hours. I went to a meeting, I came back it’s still doing it. Could you stop your stuff?”  And then I realized I'm not a team of one.  I'm a team of five.  I have to be polite. 

 

And so, please be polite with me and talk to them.  And make it known, like, “Listen, if my tools misbehave, come tell me so I can fix it for you. Please don't turn it off while I'm not looking and then just never have any tests done.” And just, that’s what they will do if you don’t listen. If they bring it up and you don't listen, badness happens. 
   

Okay.  Oh, yeah.  Using up all the resources so no one can run tests, you are an unpopular person.  It is like if you drink all the coffee and you don't make more. 

 

Okay, so next, 6, impossible service-level agreements. 

 

And so, this might sound really obvious to you because of the way I'm going to explain it, but almost every client that I work with at first is like, “We can't do that.”  And so, I like to have two service-level agreements for the whole org.  And one is all of the mistakes we made in the past, generally referred to as technical debt.

 

So, the first time I do a scan with a tool that's like a benchmark.  It is a sad benchmark, just to be clear. And I have a SLA of like we're going to fix three criticals a month, let’s say.  And then once those are done, three highs, et cetera.  And we're going to chip away at it really slowly.

 

And then I have another SLA.  Every new code thing you do, every new bug you build, you’re only allowed lows.  You’re not allowed to put a new high, critical, or medium code into prod. So, that means if you wrote new code, and it's a high, that's not okay.  That's not okay. 

 

And there are lots of tools that can automate this for you.  Sometimes you have to like use Microsoft Excel, and that’s okay.  That's my favorite security tool.  But basically, if you can have two service-level agreements, you can chip away at your technical debt and then make sure no new scary stuff gets out there. 

 

However, what almost every company I work with start with when we start a discussion is, “No highs or criticals in prod from now on.”  And so what will happen is there will be a legacy app, it’s nice, it’s old, it is almost as old as me.  It has lots of vulnerabilities, as you might expect.

 

Because software does not age like wine.  It ages poorly.  And so you have this app and it is full of tons of vulnerabilities. And then you have a person like me. And I send in like PR requests to fix a whole bunch of critical bugs.  Well they're like “Oh, there is still 200 left.  So you're not all going to prod.”

 

So, I fixed five awesome bugs and they won't let me put bug fixes into prod because I’m not meeting the SLA. And right now you’re like, “That seems so obvious. Of course, they should let you put fixes in.”  No. They don't. 

 

And so, a few years ago – so I was consulting with one client for three or four years.  Mostly so that I could do some DevSecOps every week and keep on time and sound really smart when I do talks like this because I literally did it the day before.  And I had to leave there last year because of business stuff. They’re awesome, just to be clear, it was me. I signed an exclusive contract for a while. It is done now. Anyway, and so I had to leave them. So I hired my friend to replace me. 

 

And her and I, and a year later, and that means this year, we had coffee together as we do. And she's like, “So Tanya, I was looking through the bug tracker and there is still lots of bugs open.” And I was like, “I thought I fixed these. I fixed those.”  And she is like, “So I went and looked at all the merge requests and I looked up your name and I looked up my name.  Did you know that you submitted 100 critical bug fixes when you worked there?” I’m like, “Oh, it’s not that high.” She’s like, “No, I looked it up.”  “You did?” And I was like, “Yay.  I fixed all of these bugs.” She’s like, “No, my dear, they did not merge any of them.  They did not merge any of mine and that’s why I quit today.” What? She was like, “I submitted hundreds.”

 

She submitted way more than me because she was full-time and I was part-time.  She was like, “There are almost 500 critical and high bug fixes and they wouldn’t. Because we didn’t meet the SLA.”  She was like, “I took my little coffee cup and I threw it.  And I was at home and I worked from home so then I had to wash it.  But I showed my rage.”  She is so funny.  She is really amazing. And she’s like, “So I quit.”  And they're like, “Why would you quit?  We'll give you a raise.” She’s like, “You can't pay me to be just completely frustrated every single day.” 

 

And so, when you make your service level agreements, this may sound so obvious because the way I’m explaining it, but like ask them, “Does this make sense to you?  Do you think you can do this? Is this possible?”  And when you do that and then you listen, so you don't ask, take no notes and ignore them and then do whatever you want.  You like listen to them.  You end up with things like two different SLAs.

 

And slightly more work, but then those five bug fixes I did would have been in there. And then her hundreds of fixes, et cetera et cetera, and that organization would have been significantly more secure. 

 

And apparently, some of mine did get in. Because I'm very whiney and vocal.  But she was like, “About a hundred didn't.” 
And that was heartbreaking.  And don't tell me she broke her NDA.  I knew their stuff better than they do. 

 

Okay, so, untrained staff. So, this is a thing I have mostly seen in the government.  I worked in the Canadian government for a long time.  And I have worked with the American government sort of peripherally a bit.  But I have spent a lot of quality time with We Hack Purple community members.  So, I run a little online community.  And we have venting sessions which had some house rules where we never record anything. 

 

And government employees, you are amazing patient.  I don't know how you do it.  I feel like you all deserve a pat on the back, and maybe a hug, for your patience.  And they will be given like zero training. It’s like, “You do DevOps now.  What's up?” 

And it is like, “But last week I just did regular ops.  And we were only doing okay at that.”  Cool.  Infrastructure is good.  Next week.  Good luck with that. And like no training.  No coaching.  No change in leadership.  No person that's like, “Okay I have done this before.  Let's go forth and conquer.” Like no support whatsoever. 

 

And then guess what happens?  At some point, the staff makes a mistake. Just me with DevSlop and my OWASP open-source project.  It didn't matter.  So, our entire web app project was our website for our project that mostly made fun of me. Because I built it, so I’m allowed. And that was it.  So, if it went down no one cared, right?

 

But imagine you're working in an environment and you are suddenly using Kubernetes and you just like read about it on the internet a bit and watched one old conference talk. And you did it all on your own time.  And then trying to do Kubernetes. And then, this happened to a government worker that I know,  and someone put an extra 0 by accident.  And so instead of 100 containers, it was 1,000.  And when the cloud build came in that's when they noticed. 

 

And it was a very bad day.  And luckily, because it’s not private industry, they couldn't fire them.  So, they still work there. But they’re like, “Then I felt humiliated and stupid and bad. And like I'm not qualified to do my job.” And there was a lot of like, “Should I just quit before they fire me? I'm terrible.” I’m like, “You are not terrible.  They give you 0 guidance.  They gave you 0 training.  0 leader ‑‑ come on.” 

 

And so, please train your staff.  I realize that I work at a training company so I'm bias.  You don’t have to hire me. You don’t even have to do official training. You have to provide them knowledge. 

 

This could be a mentor. This could mean bringing in someone that cost too much from another company who has done all of this before, who mentors people and has people job shadow.  Like there is are a ton of ways to train people and get them the knowledge they need to do their job.  If you are in management, and you do not need to raise your hand, you could just like think about it inside, that's your job is to keep your staff ready. 

 

Okay.  Mistakes that could have been avoided.  That's true. 
   

So, bugs lost in the backlog. Okay. So, whenever I hear, “Don't worry Tanya, it is in the backlog.”  That's why I'm worried, buddy.  That's why I'm worried.  If something is critical I don't want it going into the backlog, I want you addressing it now.  If I walked to your desk, or like found you in the online chat channel, and I was like, “Hey I need to talk to you now.” I don't want to you put into a backlog like maybe you will also do that, but I want this actioned now. 

 

And so, I feel like the backlog is a place where bugs go to die and they just stay there forever.  And I'm not saying we shouldn't use a bug tracker. Those are important.  I used to use one all day every day.  But it is really important that us security folks look through the backlog.  It is important that, first of all, every single security bug has like the little tag or the flag, or whatever you want to call it, that marks it as a security bug.  You need to make sure those don't get removed without you knowing and you need to make sure that maybe every 30 days you look through and it is like, “Wow somehow we have 100 more bugs this month than last month.  Did we do something different? Did we get a new tool?  Did we hire a super amazing tester that found new stuff? Or are things going really poorly?” 

 

And so, I try to review things every 30 days. Sometimes it slips a bit past then. But it is important that bugs don't get lost there forever. 

 

Okay. Next. No positive reinforcement. Negative Nellys everywhere.  This might sound silly but I, so like people tell me they are like, “You missed your calling as a kindergarten teacher.”  But I only like my children.  The rest are terrible.  It's true, right?  Mine are the best.  Yours, they're okay.

 

But the point is like, just like little people, like little, tiny humans known as children, really like positive reinforcement, so do adults. I'm a big fan of giving people a high five because they fixed a bug.  I’m a big fan of writing an e‑mail to their boss. So, I just worked with this dev last year named Jenny, I’m like, “Jenny’s a rock star. She fixed every single bug including the lows and got it out by the end of the week. And I don’t know if you are considering giving her a raise, but I want to like encourage you to do so.” And Jenny the next time I saw her she is like.  And then she kept fixing bugs for me. 

 

So try – so, like security folks are so known for being negative.  We come with bad news, et cetera.  What if you showed up with donuts one day?  What if you showed up with bagels, cream cheese, fruit, whatever it is, and tried being friendlier?

 

I used to work with a project manager who owned an apple orchard and she would just show up with apples.  Her name is Gail, I really like her.  And she would be like, “Hey, here’s some apples.  By the way, I need some progress report.” Everyone wanted to see her because she would bring you delicious fresh apples. And like, she had an orchard.  She had a zillion trees.  That's nothing for her, right? 

 

I’m not saying you have to bribe people to do their jobs. But I am telling you it works really well. 
   

Okay.  So, only worrying about your part. 

 

I used to be so guilty of this.  I don't know if any of you seen the meme with the little girl like this and a giant house is on fire behind her. And it’s like, “I gave my code to ops.  It is their problem now.” And I used to be that dev.  I was like, “I don't know why it is not running. That sounds like a you problem.”  If it crashed it’s not like, “My app is perfect.  It works on my machine.”

 

And so, it turns out that only worrying about our part sucks and it does not mix at all with DevOps. And so, I try to get feedback all the time. I try to talk to people. I try to make sure that they are – that we are on the same track.  And I often find like at first I’m like, “Hey do you have any feedback? Is anything going poorly? Is anything going better that you would prefer?”  And at first, they were like, “Everything is fine.  Everything is fine.”

 

Then I remember, I remember in 2021 this dev group where the previous guy was like Dr. No.  Like, to be clear, he is brilliant.  But all he ever said was no.  And he has way more AppSec experience and he has a very deep knowledge and all. And he is so terrible to work with.  And then so I show up and I’m like, “Listen, if you need something tell me.”  And they're like, “So you say you want to build trust?”  Yes.  Yes, I do.  I'm like so excited. I’m like, “Tell me what's going on.  I'm going to fix it.” And he’s like, “So your colleagues on the security team rolled out a product – which I won't name – and we now can’t access localhost.”

 

So, who here knows what localhost is? Besides a dev’s best friend. Okay, so when you code in your IDE, you like write code and you’re like, “This looks pretty cool.” And then you say, “I want you to run it.” And localhost is the power within your computer that makes the app start running right before your eyes. 
  

 

And so you don't need a server, you don’t need a container. And it will just run for you and you can go through and test and be like, “ That looks terrible.  Font size 4 was a mistake.” And so, if you do that -- so basically devs, when I was a developer, I did that a hundred times a day, literally. 

 

I constantly used localhost and they are like, “Yeah, they're doing a proof of concept and their five devs can't access localhost.” And what that means is every time they want to see their work, they have to find a dev server, publish to it, pass the CI/CD test, and get there. 

 

And meanwhile, they just want to see like, “Is this function actually calling when it should be?”  And they’re like, “It is taking so long.  We completely screwed up our sprint.  We're super stressed. Can you do something?” And I am like, “I am on it.” 

 

And so I went and I told the security team and they are like “No, it is working perfectly.” I’m like, “No, it is not.  They can't do their work. I explained what localhost was.” They’re like, “Oh, my gosh that's terrible.  Why didn't they tell us?”  I’m like, “Did you ask them for feedback?” They are like, “Why would we?”  Oh, okay.  So, “You would because then you would know this.”  And they are like, “Oh.” 

 

And so then we like made a call.  Turned out they made an error in the way they rolled out. The tool was fine. It was us, not them. And so we fixed it and two weeks later the devs are like, “We like you now. We like you.” And I’m like, “What else do you have for me?” 

 

Because if they're not able to release code, and they're not able to test the code and see if it is good, that’s still a security issue, right? And so,  we can't just worry about our part. Even though I used to like really doing that.  It is easier to do that.  It is not the right way. 
   

Okay, so multiple bug trackers. I have not run into this very many times until security.  Until the developers will all track their things in whichever one.  And I mention Jira all the time because it is by far the most popular bug tracker for English-speaking and French-speaking humans on the planet.  And those are the only languages I know. 

 

But then I got to security and they are like, “Oh, yeah, we have the risk register over there. We have our SaaS over there. We have our DaaS over there.” And they are expecting me to check like seven dashboards a day.  I'm not doing that, no.  And so, I'm a dev.  So I just wrote something to smash it all together in one. And my boss is like, “Ooh, we should hire ex-devs more often.”

 

Don't have multiple bug trackers.  Don't have a security bug tracker and a regular bug tracker. If it really truly is extraordinarily sensitive, then go speak to them about it directly.  But most of the time it is not.  Like it is like, “Oh, there is cross‑side scripting with one of our apps.” Yeah. If you have more than 200 apps, I guarantee you that at least one of them has cross‑side scripting, if not a whole bunch.  Like something around 60% of websites across the planet according to OWASP, and they know what they’re doing when it comes to AppSec.

 

Like you probably have some.  It is not a secret.  It is not the end of the world.  Like exactly where it is, yes, I wouldn't want to put it on Internet. But I trust my developers to want to know that so they can fix it, not so they could like become evil and hack us.  They have super-powers all across your network.  If they hated you, you would have a really terrible year.  So, they're not going to become evil just because you show them the bugs.  And so we want one bug tracker, ideally. Unless it is extraordinarily truly sensitive and presents real risk to our business. 

 

    And I have heard many times, “Well, but I…” And I was like, “What's the risk?  What's the actual risk to our business? Explain.”  And if you can't explain, “Sorry it is going in the bug tracker.  Bye.” 

 

I fight with the security people more than the devs when I go work somewhere.  They’re like, “It is like you are one of them.” I am like, “I am. Shhh.” 

    Okay, so the next one is an insecure system development lifecycle. 

 

So, the system development lifecycle is the methodology that the developers follow to create software.  And you can do waterfall, you can do agile, you can do DevOps, you can do waterfail, which is what I see a lot of companies doing. Where they’re like, “It’s sort of like waterfall but then we have some sprints some time and maybe we have a scrum once a month.”  And they are just like, “We are mashing the parts together we think are nice for our work.”  And that's totally fine.  But if you are doing DevOps, I used to -- I have seen a lot of security teams and if you are one of them don't feel bad.  But they’re like, “I listened to the vendors. And just to be clear, vendors are my friends. They make awesome tools that make my life better. But the sales folks, they are not my friends.”  And they will tell you, “Oh, just put this in the pipeline and all your security is done.”

 

And so I see where there is no security requirements, there is no threat modeling. There is no security architecture review.  There is nothing except in the CI/CD.  And the rest of the SDLC there is zero support.  There is no, “Oh, you are doing an API? Awesome.  We go through this API gateway.  Here is how you set it up.  You have to do this. And if you don't do this, and we find you, it is going to be bad.  So please do this. We’ve made it easy for you.” 

 

They have nothing.  They just are like “Oh, I bought three tools.  I paid $500K for them and I put them in the pipeline.  I'm done.  I'm going to sleep.”  And that doesn't work. And if you are doing that, I, in an extremely biased but loving way, suggest you read my blog or check out the OWASP project.  There is so much more cool stuff that you can be doing that costs time and not money.  That gives real serious value as part of the system development lifecycle. And I literally talk about all the time because I love the SDLC. 

 

But we are going to try to keep on track.  So, an overly permissive CI/CD. So, I have only seen this twice so far but I have heard stories where anyone can do anything.  And so, I went through and I saw all the developers had turned off all the really nice SaaS tools that we purchased and paid a lot of money for. And that we thought had been running.  I see developers who make 500 ‑‑ I worked somewhere and they had 500 pipelines and 300 of them were like one guy’s good time.

 

And I’m sure he’s a great employee, but I’m like, “Do you know that this guy actually made like 287 just for himself?”  And I was like, “So, I'm not going to secure those.  Those need to go away.  What's happening with this gentleman that he has enough time to make this many pipelines?” And so, I made him do some janitor duty and clean it up. 

 

And so, don't let anyone do anything they want.  Especially interns.  I adore interns. One of my best employees that I’ve ever hired was intern and now she is full-time and amazing and wonderful. And so, don’t let everyone have full powers on the CI/CD. Treat it like the valuable resource that it is and protect it. 
   

 

And 14, automation only in the CI/CD.  Okay.  So this ‑‑ okay let's say I want to automate everything.  So, I'm a dev.  Sometimes I automate things at home.  Like have a little hobby fern and I have automated all the watering because I don't have time for that. I just want to pick the flowers and eat the food.  And so, you can automate a ton of stuff. 

 

And I find a lot of AppSec people, they come from two backgrounds.  And one of them is I used to be a developer and I shall automate all the things.  And that's what I am.  And so I just assumed they were all like that.  Because I was like, “Why else would you want to do AppSec if you didn't build apps?  Where does your love come from to do this job?” 

 

But the other side I find is often like a GRC person who was like late to a meeting or ticked off the wrong person.  It is like, “You are in charge of AppSec right now.  Way to go.  Have fun with that Brad.” And so, as a result, they don't have all the technical skills. 

 

So, they know risk and security.  They're very bright. But they don't know the possibilities that are available.  And so they’ll be like, “Oh, I have to do it in the CI/CD because I need it to be automated.” No. Like every AppSec tool you buy you can make it run every night.  You can make it run every Thursday at 4 p.m.  You can have it when you check in the code it will run then.  You can have it only when you merge to the main branch. 

 

And when I tell them that, their eyes just light up. They’re just like, “Why didn’t anyone tell me this?” So, I'm telling you this.  You can automate all over the place.  Everything does not need to go in the CI/CD.  You can still be completely awesome, totally automated.  Obey all of the DevSecOps rules and still have people like you.  So, if you do have a scanner, and you like it, but it is really slow, let’s say it takes five hours to scan the entire code base. That’s cool. Do it on Sundays. Every single Sunday. And then Monday morning, you get there you get an email with the results, you go through it. Or you can have it auto-email the devs.  Like all of this stuff can be automated. If it has a computer near it, it can be automated.

   I have friend that like he will be in bed and he will set coffee maker to make coffee and go back to sleep for a bit. I say, “You can automate whatever you want if you are after it. And if you treat the developers really well, they will automate it for you.”

 

Okay, so, last one, hiding mistakes and errors. 

 

So, how can we learn if we don't share information? So, I have shared a bunch of information with you today.  When I work in organizations, I now share information when we fail. 

 

So, the first time that we did this, I was working at a company and we got like a very, very, very inexperienced guy to be the manager of our team. For just a few weeks, because the real boss had left and we were recruiting a new boss and none of us wanted that job. And so it was like, “Sorry you were late to the meeting.  You have to act as our manager for a bit.”

 

And so this one dev team wouldn’t do what we wanted and he insisted that we share all of our dirty laundry and all the different incidents that we had.  And I was like, “We can't admit that we made mistakes.  They won't trust us anymore.”  And he was like, “I think if we are vulnerable it will build more trust.” And I was like, “Don't do this.  This is a bad idea.  No, no, no.”

 

And then he was right.  I was totally wrong.  That team we shared the secrets with became like the champions from then on.  They were like, “We are your warriors.  This happens on our watch never again.”  And so they became like pushing other dev teams and challenging them.  They were like, “We fixed this many bugs this week.  What did you do this week? Losers.”

 

And so, if we can share even just within our own organizations more, we all get to learn a lesson instead of just that one dev that made that one mistake. 

 

And so, if I can get you to take away one single thing from this entire talk it is this, please don’t  ‑‑ I know that we can’t share everything.  But whenever you can, turn an error or a mistake into a lesson. And be humble and explain, “I'm not perfect.  I made a mistake.” And it sounds weird, but it makes people like you more.  It makes people share their mistakes with you and then you can help them fix them. 
   

And so with that, I want to summarize for you. 

 

So, what did we learn?  So, some people learn best from knowing what went wrong as opposed to, “Do this because I said so.”  They're like me where they're like, “But why?” 

 

And that's normal and that's okay.  I want you to be able to see these errors from like as they start to happen so you can stop them. 

 

I would like it if you all had amazing DevSecOps programs.  I am one of those AppSec people where I would love to put myself out of a job.  So, please just all be awesome. 

 

And these are several strategies.  So, like if you want to review the video later, my slides are online, I’m going to give you a link.  Like please steal all my ideas. When I give a talk I’m like, “Please, please steal this and do this at home.” But again, RSA is not at fault if you do it and it doesn't work.  Blame me, not them, please. 

 

So, with that, just a second. Oh, no, no, no I'm not done. I have a few more things for you. 

 

So, next week, look over your program and see if there are some things that you can fix.  And in the next three months I'm hoping you will look through your backlog and see if there are some things you can remove or update. Tune your tools. Just basically do the things that we said. 

 

I'm supposed to do a summary slide.  I'm not a summary person.  Sorry. But I have some resources for you. And all of them except the books are free.  And so, this is a PDF summary so you can just go and download all of the slides immediately. I'm also going to have this link on the very last slide. So, if you didn't get your phone out fast enough you have another chance. So don’t worry. 

 

And then also, I have a community online and every single part of it is free.  We have a few events per month online and they are all free.  We have videos, we have demos. We have a whole bunch of courses, including me teaching secure coding and AppSec all for free.  And the best part is the humans. So, there is around 7500 of us now.  All of us are nice and friendly. And we have a code of conduct. And we’ve never actually had to kick anyone out because they are all actually really great.  And so, if you want to meet more people, or have your devs meet more people, that's the place. 

 

I love DevOps. These first four books are DevOps books. I love these books. I didn’t have a chance to add this, but Investments Unlimited was written by the same group of authors. And it’s all about security and compliance and how to automate it and make friends while you do it. And the last book is Alice and Bob Learn Application Security.  And I wrote that.  And me, my mom, and my grandma said it was great.  So, my grandma has read some now. 

 

I have a podcast where I just talk about AppSec and I am silly with other AppSec nerds.  If you want to know more and like dive down silly deep rabbit holes with us, like why is this SaaS better, how exactly does it work, what the heck does stack analysis mean. Sometimes they are lighter, sometimes they are heavier. We try to do beginner, intermediate and advanced and like go between episodes like that. But anyway, we are on season 3. 

 

And the last resource is me. I'm a giant nerd on the internet.  I'm at your service to help you. This is my jam. I love doing stuff like this. I love sharing information whenever I can.  And so I’m constantly sharing videos and lessons. I'm good at AppSec.  Not very good at business.  And that's okay. 
   

 

And so with that, I want to say thank you very much for your time and attention today.  Oh, wait, wait, wait.  Oh, perfect.  I was like no, no don't take the link off.  Thank you very much for your attention today.  I really appreciate you coming to this.  Thank you.

  (Applause.)

  

>> TANYA JANCA:  We have seven minutes for questions.  But then I'm just going to stand outside.  And I have stickers.  So if you are a person that likes stickers or wants a business card, or anything like that, I will be standing outside in seven minutes from now. 

 

But for now, do we have any questions? And if you do have a question, I'm told to tell you, or ask you very politely, if you will go to one of the three microphones so that everyone can hear it instead of you.  And if you don't do that, and just want to yell it, I will just repeat it.  Do we have any questions? You need to like maybe wave your arm around a bit. 

  

>> AUDIENCE MEMBER: Hi Tanya. Great presentation.  You nailed it.  Thank you.  Just ‑‑ it is not a question, but a comment.  I want to add one more best practice I’ve seen.  I have worked in AppSec for close to 20 years.  So, I'm familiar with this.  Somehow people often associate DevSecOps with cloud. And I don't know why. Some organizations don't want to do DevSecOps because they are scared of cloud.  I’ve seen like, there is no relation to the two.  Although you can do DevSecOps on the cloud, you have the benefits of elasticity and all of that. But you can totally do it on‑prem. So, DevSecOps and cloud have nothing needing to be together.  They can run totally independently.  I wanted to add that.

  

>> TANYA JANCA:  Can I add that? 

 

>> AUDIENCE MEMBER: Oh, please.

 

>> TANYA JANCA: And I will be like some awesome guy at RSA. 

 

>> AUDIENCE MEMBER: Thank you. 

  

>> TANYA JANCA:  Thank you.  That's a really good point.  100% agree.  Do we have more questions or comments or anyone want to share anything?  Because otherwise, I will give you five minutes back. And then I will go stand outside and I know a whole bunch of you will ask me questions there.  Because people are shy, especially when there is a microphone. 

 

Okay, well, if there is no one else, thank you so much.  Thank you to RSA.  Thank you to the sound and video people.  Thank you to every person.