Jason Nelson -- Three Pillars of Threat Modeling Success: Consistency, Repeatability, and Efficacy

Jason Nelson, an accomplished expert in information security management, joins Chris to share insights on establishing successful threat modeling programs in data-intensive industries like finance and healthcare. Jason presents his three main pillars to consider when establishing a threat modeling program: consistency, repeatability, and efficacy. The discussion also provides a series of fascinating insights into security practices, regulatory environments, and the value of a threat modeling champion. As a threat modeling practitioner, Jason provides an essential perspective to anyone serious about application security.

FOLLOW OUR SOCIAL MEDIA:

➜Twitter: @AppSecPodcast
➜LinkedIn: The Application Security Podcast
➜YouTube: https://www.youtube.com/@ApplicationSecurityPodcast

Thanks for Listening!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Chapters

• 0:00 Introduction
• 2:13 Security Origin Story
• 7:48 Threat Modeling the Cloud
• 16:44 Threat Modeling Pillar One: Consistency
• 27:30 A Threat Modeling Champion
• 33:35 Threat Modeling Pillar Two: Repeatability
• 39:07 Threat Modeling Pillar Three: Efficacy
• 48:09 Lightning Round

Transcript

Chris Romeo: 0:00

Jason Nelson is an information security executive focused on building programs and directing teams in highly regulated financial healthcare and insurance industries. Leading global teams of managers with cross organizational accountability, Jason has a strong knowledge of corporate information assets, risks and threats, and effective risk mitigation strategies that enhance security and address policy and regulatory requirements. Jason joins us to share his three pillars of threat modeling success for programs. We explore each pillar, defining what it is, success factors, and examples or stories of impact. This episode is jam packed with threat modeling goodness. We hope you enjoy this conversation with Jason Nelson. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I am the CEO of Devici, a threat modeling company, and also co host of said Application Security Podcast. Robert is out traveling the Great American Plains. I don't know where he is, but I just made that up like he's in the Great American Plains right now. Uh, but I am flying solo. But I am joined by Jason Nelson, who will be our guest today to talk about what we're thinking of as pillars of a, of a strong threat modeling program. But Jason, before we even start, even give you a chance to take a breath. We like to jump right into the security origin story and get an understanding for where you're coming from, how'd you get into this world of security.

Jason Nelson: 2:13

Uh, well, thanks for, uh, having me, and, uh, I'd like to think of Robert now trying to help Yellowstone finish Season 5 if he's out in the Great Plains, so,

Chris Romeo: 2:25

Maybe he's being filmed. Maybe he is a bit part on that show. I'll have to

Jason Nelson: 2:28

be, uh, that'd be awesome. Um, yeah, so, my start, I Was doing like a lot of people do as they're starting out. Uh, you do a system administration, you're, you're working kind of everything type jobs, you know, you're the, you're the IT person. Uh, I got into financial services early on in my career and that was obviously security is first and foremost for that. And so. I started, you know, diving in on training, uh, got my certification, ICI SSP, things like that. And that kind of led to jobs where I started doing pen testing. And, uh, pen testing was really, really exciting, uh, as far as just getting access and learning all about how to. Execute, all these things people talked about, vulnerabilities and CVEs and that was back in the days where you still could get onto modems and get into the network, not so much anymore. But, um, that all kind of evolved into kind of bigger and bigger roles through security and building out security programs. And so. Today, uh, building threat modeling programs and, uh, working in security architecture.

Chris Romeo: 3:39

Okay. And I think it's helpful just for our listeners to have some perspective. When I think of what you're doing, I think of you as a practitioner. You're somebody who's running a program. You're kind of in the trenches, and I think that's important to, to make sure people understand because when we start to talk about what makes a good threat modeling program. You're coming from the laboratory of building a threat modeling program. You're not just somebody who has theories, you're somebody who has theories and you're putting them into action and trying them out and modifying and changing things, right?

Jason Nelson: 4:13

Yeah, I've been lucky enough to be, uh, at companies and put in positions where I can threat model. Uh, I get to dig in and play and learn and, uh, kind of create that craft and cultivate it. And so at a couple of different companies now, I've, uh, been working on these types of programs and what works, what doesn't. Uh, and specifically what, you know, what works for these types of industries and at what scale. And because every company has a different culture, every company has a different requirement and need. So, um, how to make that threat modeling program work in those, uh, environments has been, uh, kind of my growth path. And so right now I have a fairly decent sized threat modeling team that works with me, but I myself still threat model.

Chris Romeo: 5:03

Nice. So when you think about all of the experience and different jobs that you've had in different jobs leading up to focusing on threat modeling, what do you think are some of the core things that have prepared you to be a strong threat modeler. I know you mentioned pen testing and some of the other things. And, uh, so, so like what, what steps did you take along the way that, that built this threat modeling foundation for your career?

Jason Nelson: 5:32

I mean, I think I've benefited from hopping and kind of being in front of some of the different evolutions in IT over the last 20 some years, and having those experiences kind of help you recognize that, you know, all technology is kind of the same eventually, it's just a new iteration. And so you're looking for the same types of things. So, uh, being on the Early part of, uh, pentesting and where you're still building out your rig, you know, Linux, you're, you're configuring the kernel, you know, it wasn't just, hey, I'm putting in my USB stick and running and it's done. Uh, those types of things made you really understand how the system works and how the, the configurations impact how you can interface with the system or abuse the system. And as I worked in, you know, I moved into one of the large cloud providers and worked for them for several years. And that was another evolution where, okay, well, these are just computers that somebody else owns and they're providing me services in front of that. So understanding that there's still bits and bytes running and there's still processes and there's still other things. How can I systematically. Uh, you know, understand how to ask questions and, uh, assess those things and then try and abuse them if, uh, to discover, you know, new vulnerabilities. So that, that type of process, uh, and different roles have given me a good background and wealth of knowledge as far as the different types of technologies. Um, but ultimately when I interview people, uh, I don't necessarily even expect them to have a real huge, broad base of knowledge like this. It's. How inquisitive are you? Because I think for threat modeling, that's one of the most basic things that make you good at this job versus someone who is excellent at a technology. Because someone who's excellent at a technology may make assumptions as far as, well, I know how this protocol works, or I know how this configuration works because I've done it a million times. But if you've never thought about it in the sense of, well, I don't know how it works, so I'm going to ask all sorts of questions and that's where the abuse cases start coming into play. And that will be the difference between an expert in a field and someone who is just really inquisitive and threat modeling.

Chris Romeo: 7:48

So you mentioned cloud providers and, and some experience in a cloud provider in the past. I have this, I don't know, maybe you can correct me if I'm wrong here. I have this, I have this idea for what the threat landscape looks like for a cloud provider. And tell me if I'm right or wrong here. Is it primarily a standard set of threats that you see kind of over and over again, or is there, is there the opportunity to freestyle in a world like that? I'll give you an example just to kind of set the stage. So I worked at Cisco for 11 years. I rolled out threat modeling as one of the things that I did. I, uh, you know, was teaching people about threat modeling a lot. And, and one of the fun things about Cisco was. They still do make products in every category you can think of, but in those days I think there was even a wider depth and breadth of product spaces. So we were constantly looking at, you know, an IOT device, a backbone router, a phone, a wireless access point, like they're all technology products, but they were all in different places providing different functions. And so the threat landscape was gigantic because we were thinking we were applying some standard threats, but we were also dealing with different technology types that were in different environments and things. But when I think about the cloud, it seems like it would be. There would be more of a standard set of threats that we could work from, but I've never had anybody I could ask that question to until now. So what are your thoughts based on your experience?

Jason Nelson: 9:13

Um, I have a couple different answers for this depending on how long of a format we have. Um, one of the first things is yes, they're going to have standard things because those, they're still, they're still using, you know, Linux, they're still using the hardware and obviously some of them are starting to build their own processors and storage and other things like that to try and get away from, you know, people being able to discover and reverse engineer it and things like that. But at this, they haven't changed physics. They haven't changed, uh, computer science, you know, so yes, they still have those things and they still have those challenges of patch management. They still have the challenges of updating, uh, Out of, uh, support technologies, the dip, the two, the big differences from the consumer perspective, or, you know, from the platform itself, and that's what, uh, some of them refer to as the shared responsibility model. So you'll see in a lot of research papers out today that people are finding through fuzzing through other, uh, capabilities of discovery or testing that there are vulnerabilities in the production implementations, the customer facing services that they're selling. Absolutely. It's the same techniques as they would use on, you know, in your data center or in your house, even, you know, so all those things exist. And the big thing, I think a lot of customers need to recognize, because just because the vendor's telling you, don't worry, it's secure. We spend a lot of time working on that. They're selling you something. So you need to be a little bit, you know, untrusting there and recognizing that they're putting stuff to market as quickly as possible. And we are all beta testers. And that is why we threat model. That is why we ask the questions and test it.

Chris Romeo: 11:01

And the shared responsibility model is basically pushing some of the requirements on us as the consumers. Now granted, they're, they're saying, yes, we provide some infrastructure level security things and you can go read their security policies and their security pages and it says how they have all these different, you know, guards with machine guns at the data centers and everything like that. But the shared responsibility model is as much pushing it, pushing some responsibility back across the table to us as consumers, as it is them, you know, putting together a, a secure infrastructure inside the data center.

Jason Nelson: 11:40

The easiest way to put it, because I mean, a lot of people talk about, Oh, SOC 2 and other types of assets that The cloud providers provide you to demonstrate their compliance to frameworks. You might not be beholden to that framework. You don't even, some people that are less, not in regulated industry or smaller, don't even know what those are or care, but they do know at their business, the thing that I'm using your services for is important to my business and the loss factor of that could put me out of business and that right there is the difference between what a vendor does for themselves and that shows responsibility towards the customer. And so if your loss factor is so huge that you'll go out of business or it would be they're irrecoverable. You probably need to spend a little bit more time looking into the configuration and your choices on the equation of, if I do this, could someone come in there and make a change that would create that loss event? And that, that's exactly the focus people need to start thinking about, about cloud.

Chris Romeo: 12:45

Yeah, yeah, I wonder if we'll ever reach the point where we have the standard set of threat models that come from the cloud providers, and then we can build on top of those. Like, I mean, that would be, that would be pretty enlightening for building a threat model for something that I'm building. a product of my own inside of a cloud environment to be able to layer on top of things that they've already explored. And I know that's a, that's a big question that's, that's happening in the industry right now. CISA has had some, some, I've seen some things floating around about, you know, sharing the threat model. We talked about it on another podcast I do, Security Table, you know, sharing the results of the, the, the, some of the data, the, the information from the threat model. But for me at the end of the day, like I don't have intellectual property in my threat model. I don't know that I'm going deep enough. I don't know that I'm doing a good enough job making, you know, spreading threat modeling across everything that I build. And so I don't know about sharing, but it just seems like with those cloud things, they could build us some standard stuff. They've got the reference architectures, like a reference threat model set would be something that would be pretty cool to see.

Jason Nelson: 13:52

Now, that'd be very cool. Um, give a shout out to some work I used to do, uh, uh, at FINOS Project, and it did exactly that, uh, as far as how do I, as a regulated entity, uh, adopt this cloud service. And it was building out the controls in like all the way to implementation and testing. So it had Gherkin, it had, uh, your security test cases so that you could verify, not just assume if I configure this this way, I'm okay. It's like, there's. testing methodology so that you can bring it to the level you need to bring it to. Uh, that's something I started about six years ago. I haven't really participated since. A new project that I'm working on now is more about portability because a lot of regulators around the world in finance are talking about cloud portability. And so the project, again, with FINOS is towards what you were saying about that standard set, is we're actually working with NIST to chain and use OSCAL, and we're going to put in a new section, uh, that has this test case. And so that way, that is kind of like the standard way to verify, uh, whatever NIST thing that you have. You can see this is the test case to verify that. So it's not just, I know I need to do this, and they're recommending the setting be this. You can actually, for cloud and other things like that, verify that this setting exists or go to cloud vendor and say this is what exactly I need you to tell me that you are doing or aren't doing and how. And that, that's what that project is about.

Chris Romeo: 15:32

And you mentioned OSCAL or something like that. I'm not familiar with that. What is, what is that thing you just mentioned?

Jason Nelson: 15:38

So anybody that's familiar with NIST and all the SP, uh, documents, uh, that say this is how you harden a server, like, um, you know, the Windows, you know, SP, uh, 53 something, you know, whatever. Everybody's. Probably heard of those or seen those eventually, um, those were published and they're Word documents and they're awful to maintain or, uh, programmatically make use of, like a human has to look at that, then translate it and do something with it. So, OSCAL for all of what NIST does is what the government said, I need everything to be machine readable. And OSCAL is that implementation. so with a machine rate, readable structure, think JSON, think other things like that, that just structured data, uh, XML, things like that. So, um, that's all it is. It's just a machine readable representation of the special publication documents.

Chris Romeo: 16:34

Something I didn't, uh, I've certainly seen plenty of NIST SPs in my lifetime, but I did not realize there was a machine readable, uh, approach to it. So that's, that's very helpful. Well, we better start talking about threat modeling programs, because that's, uh, ultimately what we wanted to talk about, but it was, it was fun to explore some of these other things as well. So, we've got three pillars that we, that you've laid out here for us as far as, pieces of threat modeling programs or things that, that are, that are, that are helping threat modeling programs to be successful. And so it starts with consistency. So I'd love for each of these, let's, let's work through each one separately, but starting with consistency, if you would, let's just have a definition in place of what you mean by consistency in the context of threat modeling.

Jason Nelson: 17:25

Consistency is definitely about, I want to be able to, uh, start an end and have a similar approach, uh, a similar result and a similar timeframe, a similar cost. And that is the consistency. Cause from whether you're a small startup or you're a large enterprise, uh, there's going to be different needs for the consistency. Um, but it is just that specific thing of, I can start and finish this and have a realistic expectation of what it's going to cost me, how long it's going to take and what the outcome is going to be. Because if, let's say you weren't the amazing threat modeler you are, and, you know, I was, uh, just like a month or two ahead of you, the differences in experience, I might produce an artifact along the way that you didn't, because you haven't reached that maturity level yet. And then a third person comes in that's like super mature, and then they're doing several artifacts. So whoever has to consume that now goes, what do I do with all these different outputs? And so that consistency really plays into some of the other pillars as you mature, because you need a common output and duration and cost, uh, because one, people, if you're in a larger enterprise, won't subscribe to your service and you won't, you know, commit to you, uh, smaller companies. And I run into this a lot, honestly, their, their decentralized approach where teams do threat modeling, but it's independent and they do it very differently, different methodologies, different outcomes, different artifacts, and, you know, different costs. Um, that ultimately leads to a problem of how do I represent, uh, to the large companies that are going to consume me, like, uh, the regulated industry. Show me your VTM reports or show me how you resolve and identify threats. And if you're decentralized, they don't have a consistent process, you're not going to be able to generate that data easily or at all.

Chris Romeo: 19:38

So this consistency then is consistency within one organization. So a startup may have a different standard that they are consistent with versus a large enterprise that has potentially a threat modeling team and, and threat modeling consultants. Whereas a startup may have one security engineer and they're pushing developers to help create a threat model for their platform. So can, so. So these, there can be different levels in your mind, though, of consistency, like the startup, you know, you're not saying the startup has to match up with what the enterprise is doing.

Jason Nelson: 20:14

Oh, not at all. I mean, everybody's going to find what they need to be consistent in. And it's all driven by the requirement for why they have a threat modeling program at all. So if they exist because they have regulatory requirements, well, that's going to drive what you need to do and produce each time. You might evolve and expand what your process is so you're consistent in doing that. At first, you're narrowly focused on, I just need to produce this outcome that the regulation requires me to do. For smaller companies, it may just be, we found that we need to do this, uh, to show that, uh, We don't have security breaches in our products. And someone in management said, that's a good idea to go ahead and go do that. Well, the consistency there might just be, all right, we'll do these three things and everybody do that the same way. And that's fine. It doesn't have to be, I'm producing an artifact that gets submitted to an external entity. It's just, I run a code scanner. I've asked some questions that are like, is this going to be external facing or internally facing? What type of data is going to be in there? You know. Is this the lunch menu or is this our, you know, core of data?

Chris Romeo: 21:31

Yeah. So is it consistency of output from the threat modeling process? Is it the process itself? Is it both of these things? Like, what is, what's the scope of consistency?

Jason Nelson: 21:47

Yeah, so the consistency, I think, in my mind is just the, you are doing the same thing again and again. And it, and whatever you're doing in that is the same. Uh, cause I think this branches off into other things like tooling versus process. Uh, and where you get value for threat modeling, what needs to be defined first. But if your consistency is literally. I've bought a tool that does threat modeling for me and gives me recommendations. Because that's all I, I don't have people. I, I, I don't have that experience, but that's what I do every time. That can be consistency. As requirement grows or something else changes in your maturity or capability demand, you might need to define a process to feed into that or to improve an outcome or to integrate with something else. Like that's the evolution.

Chris Romeo: 22:38

So the, so if we were to divide this into three levels. Uh, like a maturity model. Level 1. So level 2 might be, you need, of consistency is you have a process. Level 1 would be you have a common output of threat modeling. Even if you don't have a threat, a process. I don't know what level 3 would be.

Jason Nelson: 23:00

I would, yeah, as you get to level three or above, I mean, you're definitely getting into the quality and the depth because. If you're just starting out, uh, you have a process and that's as far as you've gotten, you might not have created loopback, uh, into other teams, or you might not have purchased, uh, threat intel feeds to inform your process. Uh, you might not have reverse engineering people that are going to say, Oh yeah, this is there. You should be aware of that.

Chris Romeo: 23:31

Yeah. Yeah, I mean one of the, one of the things I did early on in, in rolling out threat modeling was to just, acknowledge that not all threat models are created equal. But the process of people doing threat modeling is the positive cultural movement forward. And some threat models are going to be amazing and make you fall out of your chair and go, Wow, this person knows more about this than I do. And some are going to be like barely able to have threat model written on the title page of them and not be like, But the point is they're all moving our suite of things that we build towards a more secure and private future. And so that, that consistency is important, but there's also a, a measure of there's going to be different out, there's going to be different abilities and outcomes and things, but everything's every step forward, no matter how small, if it's a tiny baby step or a giant step is still a step towards where we want to get to.

Jason Nelson: 24:32

Yeah, I absolutely agree.

Chris Romeo: 24:33

How about success factors then? So like, what do I have to do to ensure that I have this consistency?

Jason Nelson: 24:43

Understand what your requirement is, why you exist, why the threat modeling program exists, why you started it. And that could be the measurement of, am I hitting that every time? Is there a gradation of what that thing is that I produce that can tell me if I've done a better job than others? Uh, easy things to share there is, on the low maturity side, and I'm running just a couple tools. Did my code scanner run? That's the first. You know, did I succeed? Uh, what, how many of the things that it detected are actionable or, you know, even relevant to what I'm doing? Cause that could be the quality of the consistency. Cause if I spend 80 percent of my time just getting rid of nonsense or noise, um, you know, it's not a super effective process. Uh, and then the more mature side of things, uh, have I addressed the scope of the population of threat models I need to achieve, uh, in my organization? Um, am I getting input from them that is, uh, valuable and usable? Meaning, if someone tells me, uh, this is the shape and size and configuration of my, uh, thing I'm threat modeling, application or infrastructure or whatever, but it's, You know, 100 percent non usable, uh, that also affects my consistency of my delivery time. And then finally, the, the outcomes, uh, of that is, am I informing and affecting positive change? So not, not just, um, I've produced a report or better, I've translated my controls into work tickets that get assigned to engineers that get fixed. Am I seeing this stuff again? Right. Because if I'm telling you about something, like when I was doing pen testing, I know I would come back to customers year over year with almost the exact same report. Like, you haven't done anything since last year. Like literally all this one, you just needed to update the cert. The cert is still expired.

Chris Romeo: 26:50

You're going to get me all fired up here. This is one of my negative takes on pen testing is that scenario you just described. People pay for the pen test because it's a regulatory requirement. They get the report and then they don't do anything with it. It's like, why did you waste your, just tell the regulators to take a hike if you're not going to do anything with it. Like it's, it's, it's security theater is what Bruce Schneier created that term a number of years and years ago, but that's really what it is It's security theater going through a pen test so we can check the box that says we went through a pen test But I'm about to you know, if you remember Dennis Miller, I don't want to go off on a rant or anything But

Jason Nelson: 27:28

Oh yeah. Love that guy.

Chris Romeo: 27:31

uh, do you need a threat modeling champion? to be, and I'm going to step outside of, I'm going to, I'm going to break my own rule. And I'm going to talk about all, talking about all three of these things, which are going to come to repeatability and efficacy in a second, but for consistency, repeatability and efficacy, these pillars of a strong threat modeling program. Do I need a threat modeling champion at the center for these things to be successful?

Jason Nelson: 27:52

Yeah. I mean, uh, anything you do at any company or anywhere, you need to get to people's focus. And understanding, because you're coming at them with a new thing, potentially work that they need to do, or at least pay attention to. And most of the time people don't give you their full attention, and they're not comprehending what you're doing. So that security champion is not only going to grab the attention of the people that need to pay attention, but they're going to help translate and inform and educate the value of this is what this is, and this is why you need it. And this is how we'll make your life easier going forward. And that is the true value of a security champion or threat modeling champion.

Chris Romeo: 28:39

Okay. And then with consistency, we have examples or stories of impact. Um, I specifically, I can, I can think of one from the days of old, but I thought I'd give you the chance to go first.

Jason Nelson: 28:51

Um, I'm sorry, the security champion or security stories from days of old. Is that where we're

Chris Romeo: 28:56

No, no, no. Yeah. So from a, from consistency perspective, sorry. So from, yeah, like is any examples of how you've seen consistency work in your, or not work, or things that made the wheels fall off from a consistency perspective?

Jason Nelson: 29:10

Oh, yeah. I mean, it's easier to go with the horror stories than, you know, I get the ways that it worked well, because usually nothing goes wrong when it works well. Um, the, from a consistency perspective, it's about integration. Um, so, uh, if you've ever did a denial of service against yourself as a company, because of a security fix, you'll realize, Yeah, consistency in process needs to happen. So if someone went ahead and did, uh, oh yeah, they get really excited, really on, uh, what was going on with a security, uh, assessment that was provided to them, went and fixed something immediately. Now the consistent process typically would have been to understand the scope of the change, meaning who else is involved in this system. And so once you shut down a capability, a window service, that another system that relied upon to deliver documents to an external party, well, that that's a problem. And so that consistency in process of, I've told you about a bad thing and how to fix it. You didn't follow the process and that inconsistency created a, uh, denial of service. Uh, those are always, always bad things in threat modeling, uh, denial or not, not denial of service. In threat modeling, an inconsistent process, uh, can erode trust. Meaning if I give you, uh, if I get a report from you, Chris, and I'm just like, this is incredible. I'm, I really liked this threat modeling idea. And then I come along and deliver something that's just shoddy. and doesn't have the same content, is unclear, is not really actionable, they're going to go, I don't know about this. And that erodes the trust in the whole program.

Chris Romeo: 31:04

Yeah, my quick story of, uh, consistency, one of the things that we did at Cisco when, uh, we were in the process of rolling out threat modeling was we were still using, um, feature specs and system functional specs. As, as design documents in, uh, you know, it was a still kind of waterfally approach to building products. We added a threat modeling section to the feature spec so that if you were, if you were writing it, when you went to grab the template, there was now a threat modeling section. And so we integrated it into at least the view so that people at least had to delete it and then make an argument for why they deleted it from the template when they turned it into their actual work product. And so that turned out to be a, just a good way to, at a really big company, inject a little bit of consistency by. Playing the process, you know, the, the product process game, uh, which in a startup doesn't exist. It's ask, ask the person sitting over there. They'll tell you how we build this product. But in an enterprise, that was, uh, that was a way to get some, some impact on the consistency side.

Jason Nelson: 32:11

From a larger perspective, let me give you a better example then. Um, I like, I like where you, where you went with that one. Imagine a process that you've built a good input, or you thought you've designed a good input into your threat modeling program. Like, I need things. Shaped and sized in these formats so that I can consume this and move forward. So I assume a level of data quality. I assume a level, level of, you know, comprehensiveness of the information you're providing me. What we did is, uh, I mean, I don't always love this. I really don't like it at all is the measurement of, uh, delivery and delivery time and those types of, uh, effort that we had to do because everyone was complaining, Oh, threat modeling is taking too long. It's slowing us down. And so I had to go through this exercise to show them. No, no, no, we're delivering. Once we get the content that we needed, we're delivering within the time box that we said we would. But, what you're complaining about for these things that took too long is because we went back seven times to get that documentation filled out properly so that we actually could threat model this. Because if we don't, if you send us an idea, I can't threat model your idea because there's no code. There's no configuration. There's no architecture. And so that's the consistency part. So it can really impact your delivery success.

Chris Romeo: 33:34

That's a, that's a good example. Alright. Repeatability. I feel like we kind of maybe introduced repeatability during our consistency conversation.'cause we did, I, I, I'm now looking at it going, wait, why did we, why did I ask about process under consistency when we had repeatability right in front of us? But from your perspective, what, what do you define repeatability as

Jason Nelson: 33:54

Yeah, this is, um, Can I do it? Can you do it? Can someone else do it and get the same outcome? So it's not just enough to have a process written down. It's about the ability to reproduce the same outcome. Uh, so if I threat model the same scope of work and same content as you do, we should be very, very close in the artifacts that we produce at the end of the process. And that's,

Chris Romeo: 34:20

you mean, do you mean you're, you and I are gonna come up with the same threats?

Jason Nelson: 34:25

we should have the same, close to the same threats. We should have close to the same, uh, reference material, and the same control recommendations.

Chris Romeo: 34:37

So how do you factor in the value of brainstorming in the threat modeling process? Because that's one of the things that, that I preach, right, in, in, in my approach to threat modeling is, one of the things that, that is a draw for Developers, for example, to participate in the threat modeling process is it's not just run this process, go down this checklist, and then it's not just a paperwork exercise. We can jump on to a live collaborative environment, and I can challenge them to think of the worst possible thing that could happen to this product you know better than I do. And it's amazing what they come up with, though, right? They're like, uh, well, this could happen. And so, but I think of that as more the brainstorming side, so there's, there's diverse backgrounds and diverse perspectives and diverse knowledge coming together so that you, you and I, if we threat model the same thing, we're probably, I'm going to come up with some things you didn't think of, you're going to come up with some, I'm not thinking of, but how does that play into repeatability? Does that break repeatability in your mind?

Jason Nelson: 35:37

I don't think it does. I think that, uh, provides an opportunity to update, uh, the prior art that would be part of your process to reference. Uh, the frameworks that you use to reference, so let's say MITRE ATT&CK or the other MITRE frameworks or OWASP or, you know, those types of, all those things should be part of your process to reference to make sure that you are fully considering all the potential things that exist already. The, this, the meeting that you're having that you just described, that should be part of your consistent process. So that you have the opportunity to at least discover that and listen to the people who actually built it. Cause you're right. They're going to know way better than you looking at code and, you know, some paper about what it is this thing actually does.

Chris Romeo: 36:31

I always, I mean, I always tell people what makes me a good threat modeler is I'm just not afraid to ask questions that I don't know the answer to. That's, that's what it is, cause like, I'd say half the time I probably have an idea what the answer is to a question. Sometimes I'll ask a question, and I don't even know if this makes sense. I don't act like it in the moment. I act like I kind of know what I'm talking about, but I'll throw a question out there just to see, can I get other people in the room to start chugging on it? And then, you know, they see the little thread and they start all of a sudden they start pulling it and the whole sweater's unraveling because I threw something onto the table that seemed like it could have been irrational almost and doesn't even apply to our product, but yet they can pull on it and go with it. So I'm glad to hear that the, uh, you know, kind of what I was describing from a brainstorming perspective is still, it still fits in the repeatability. It's having that level. of, you know, that, that brainstorming piece built into your, your repeatable process ensures that you're, everybody's, everybody knows that's an important step in how you threat model.

Jason Nelson: 37:34

It absolutely is, but that, yeah, I, the repeatability there of how they do that is because they're following a preconceived process potentially. And because if someone doesn't know to do that and they don't do it, then you're going to diverge into a different outcome altogether.

Chris Romeo: 37:53

Okay, how about some stories of any story of impact you can think of an example from repeatability?

Jason Nelson: 38:02

Yeah, yeah. So, um, we have a, I I've. My experience, I've built a couple of training programs. And so people that have been doing roles that are similar to what the new threat modeling program is that I've built, uh, start to bring their old threat models with them and they re threat model it with the new approach and process and all that stuff. And the most exciting times are when they discover new things. It's, and it was because they didn't think to ask the question. They, they had made assumptions, uh, about, oh, well, they said they have TLS. So I assume then this and that, but when you start digging in, they realize, oh, pinning doesn't actually apply here. And so therefore that token can be used here, here, here. And now your authorization model is totally broken. And it's just because of the process led to the, that line of questioning and it discovered that problem.

Chris Romeo: 39:08

Okay. What about efficacy? What's, give me, give me a definition of efficacy. This one is of the three. This is the one that I was kind of like. I don't know that I fully understand this one going in, so give me, give me your definition of efficacy here for, for a threat modeling program.

Jason Nelson: 39:24

Uh, efficacy is, can you prove it? Um, did you actually do the thing you said you were doing? And did you do it well? Um, from a security controls perspective, uh, like guardrails, everybody talks about guardrails. Uh, we built that where it's not just. I'm linting code. I'm building the security test. I'm using the Gherkin language as I describe the control to my security engineering team so that they clearly understand the behavior use case that I want to avoid or enable. And so the efficacy testing is through that behavioral testing and it shows the positive use case and the negative use case outcomes. And that feeds into our, you know, SOC and things like that of this, this is the log events that are for the positive use case and the negative use case. And this is the detective control and we build out preventative controls or in some cases where you can't do those easily. It's just, hey, when you're designing this thing, uh, just don't do that. So like, uh, when you, this is going to require you to create a credential before it even, even makes it into our system that stores credentials or something like that. Don't put it here, don't put it here, because then you expose it, right? Those are the types of things that we get into with efficacy. So from a threat modeling program, how do you demonstrate efficacy? That's the, you build in loopback processes and process integrations, so that you do a threat model, and typically you want to be early in the process, that you're not doing it after something's deployed. VA comes in along the way and does their test. Build a process to link into them, and they can show you the path to execution of the things that you thought were real and loop back and say, those were actually real, or, eh, this one didn't actually play out. And that helps you focus your control writing.

Chris Romeo: 41:21

So in your experience is, are you able to get to nearly 100 percent of coverage there? It just seems like that would be a lot of work coming out of the threat model to get to test, a test complete level, even above 80 percent of the threats that I brainstormed or wild, crazy things I came up with in the threat modeling process. So what do you see as the percentage of coverage that is acceptable?

Jason Nelson: 41:45

That's completely dependent on your use case. Regulated industry, you're going to have to report because you have a VTM program and that is reviewed and externally verified by your regulators.

Chris Romeo: 41:59

What is

Jason Nelson: 42:00

going to, Oh gosh, I'm forgetting what the acronym says, like vulnerability, threat management,

Chris Romeo: 42:06

Okay, got it. I was, yeah, I, I was hoping it was threat modeling and it was something I'd never heard before and I was like, I hope I'm going to learn a new cool threat modeling acronym, but that's okay. I'll take vulnerability and threat management.

Jason Nelson: 42:15

But I mean, it's your patch management program, essentially saying I've identified things and I fixed them. And those should reconcile and so everybody's going to have their own lever of how many days or time or how many patches or, or whatever their measurements are. That's going to be your threshold from a threat modeling program. Um, We tried to enclose as many, like, I think we've tried to get to 100%, the challenges with that are maturity, meaning when you first start out, you're going to have a lot of stuff that you discover, and then you're going to get to more sophisticated problems. Uh, an example, I don't want to allow this when this condition is. So to create a control like that, you need to build infrastructure that one is made aware of and can store in memory those things that if comes from this, okay. So that could be accounts or people or whoever, then do this. So that requires like a lambda function or other, you know, a step function or something like that. So that infrastructure is a higher maturity point than I've just identified controls and working, wanting people to work on them. So it depends on where you're at in your capability, how much, how tightly coupled your engineering teams are to your threat modeling program and stuff like that.

Chris Romeo: 43:40

And so in this model, you're, you're threat modeling the same thing over and over again, maybe iterations of it.

Jason Nelson: 43:48

Yeah.

Chris Romeo: 43:49

Okay. Cause like, I guess philosophically I prescribe Something that, you know, Izar Tarandach said it went long before I ever did, but, but threat model every, every user story. is, is something that I aim for in my threat modeling program. And, so that's why I was, I was kind of wrestling with the numbers of trying to get to a test coverage perspective coming out of threat models, if I'm threat modeling every user story, so I'm always And I guess there would be some patterns of things. There's not, it's not like you're creating something innovative on every new feature that comes out. Um, but it still made me think like that would be, that would be why getting to 100 percent testing on everything at that velocity would be. would be more of a challenge. But if you're in a world where you're threat modeling the same thing, and maybe the first time you threat model, you get to 25 percent coverage, and then in a, in a revision 30 days later, you get to 40%, and you're slowly clocking it up until you get to an automated threat model that's close to 100%. I could see how that would work. And that maybe that's, maybe that's the difference between regulated environments and non regulated. I have never played in the regulated space, maybe on purpose, maybe not, I don't know. But But it's, but that may be that kind of different mindset that, that where I was kind of seeing a void is, I think, because I just don't understand regulated as much.

Jason Nelson: 45:10

Yeah. And once you get to scale and you have tens of thousands of applications and on a ton of infrastructure, you have to define your scope of what you can actually cover. Because otherwise you're always going to fail. You're never going to meet your objectives. Uh, so for a lot of what I'm working with, it's, uh, within a scope. And that's how we can keep up with that. Uh, and the application and infrastructure are done differently. So when you're, I think you're talking about user stories related to application security. Yeah? From the infrastructure side, especially cloud, it's a lot easier to front end that and do it once and then just monitor for change rather than on a time based, because time based, just like I learned that during VA a long time ago, you can't keep doing everything over and it just keeps growing and then you have such a backlog that you can't take on anything new without hiring tons and tons more people. So from the threat modeling programs that I've built now and am working on, it's trigger based, so we have Intel that tells us when an API has changed in a cloud service. We have Intel that changes, or triggers when code changes on specific things that we think would impact the security. And so those are the ways we've reduced our scope and stay informed.

Chris Romeo: 46:30

hmm. Yeah, that makes sense. Um, how about any stories or examples on the efficacy side?

Jason Nelson: 46:39

Oh, yeah, this is, we push back on, uh, have pushed back on cloud vendors a lot where they say, Oh, if you make this setting, then it's encrypted end to end. And so we would do behavioral testing and demonstrate, Oh, that's not true. And so you fight with the vendor and fight with the vendor, and then they finally believe you. And then they go, Oh, yeah, well, that's on our backplane though. I'm like, well, I don't care where your backplane or on the internet, I don't, you don't want it exposed.

Chris Romeo: 47:08

back to that shared responsibility model. I don't trust your part. You know, you don't trust in my part. I'm not trusting your part either, though, on the back end.

Jason Nelson: 47:15

Yeah, exactly, man. That's it. That's it. Um, we've had lots of examples of that and that's because of that efficacy testing. It's not just linting code. It's performing the test. I need this to be encrypted end to end. How do I test for that? And so you build tests that are going to be accessing data and looking, is this readable?

Chris Romeo: 47:36

So yeah, consistency, repeatability, efficacy. These really are three strong pillars of a threat modeling program. And uh, yeah, it's, it's, it makes sense to me. It's resonating with what I've seen in the past, both from the startup environment and the gigantic company environment. These things, like you described, they, they fit in different environments. Just, it's going to look different, obviously, and depending on the size and the way that the threat modeling is being deployed inside of a company. So, um, let's do a little lightning round. Normally this is Robert's time to shine and take you through. Hopefully I can stumble through the lightning round questions in Robert's absence, but. How about our first one? What's your most controversial opinion on application security and why do you hold this view?

Jason Nelson: 48:31

Uh, I've had arguments with lots of people about this. Uh, I don't know if it's that controversial. Uh, but it is that app-threat modeling and infra-threat modeling can be the same process. I know this has upset a lot of people that I've introduced the program to, and we've come to the other side of that, and we're in agreement as far as execution, but, um, there's still always little things that pop up. Like, oh, but this is different.

Chris Romeo: 49:03

Yeah. I mean, I would think that would make sense to have a single process that drives both of those things. Like we can threat model anything, right? Like we go to the airport and you're walking through TSA, you can threat model the TSA experience. Just one thing I've learned is don't tell them about it. Well, you're in the line. They don't find it funny at all. Bunch of people gather around, they're like, look at this guy. Okay, so how about a billboard message? If you could put a single message on a billboard at the RSA conference or a black hat, what would be the message you would send to our industry?

Jason Nelson: 49:36

Uh, I mean, BlackHat and RSA are very different conferences. Uh, BlackHat, I'd say turn off your phone. Didn't, you shouldn't have brought it to this conference to begin with. Um, RSA, I would say start asking your people what's important. Because if you're at RSA and you're a security leader, Uh, you're running teams, you're running budgets and other things like that. You're may not be in the weeds doing the thing. So ask your people, what is the trend? What's the undercurrent? Because it's difficult to keep up with all of it. And I'm always impressed by the new things, new members on my team bring or, uh, other team, you know, they're just, they're really focused in doing research. And if you're a more of a senior executive, you're, you're not going to have time for that necessarily.

Chris Romeo: 50:31

That's true. I would, I would change a lot of the scope of RSA if those leaders all came informed. But that's, yeah, that's good advice. How about recommended reading? Any top book recommendations? Why do you find them valuable? Doesn't have to be security, doesn't have to be nonfiction. That will take something from any, any category.

Jason Nelson: 50:54

Um, I don't know, sometimes, uh, I, I've just really like, uh, deep on some of the stuff. So, uh, Mike Lye, uh, has a bunch of books on, uh, reverse engineering. And if, if you just really want to go deep and learn about techniques and tactics and, you know, just some anecdotal stories, he has got a couple of great books. Um, And then, uh, just going about threat modeling. I mean, there's, there's actually several threat modeling books out there now. And so, uh, any, any of those, um, would be something I, I recently that I read that I, I do like, mm-Hmm,

Chris Romeo: 51:40

How about a key takeaway as we wrap up our conversation here, Jason, what would be a key takeaway you'd want to leave our audience with?

Jason Nelson: 51:50

um, understand your audience, uh, from, from a threat modeling perspective. If you're looking to build a program, just the first thing to do is understand why. And I think a lot of people don't start there. I, I know I didn't. Uh, I, I was asked, uh, hey, we need to figure out why, uh, we keep having all these problems, like, why do we keep having issues with released infrastructure or applications? And so, well, you need to ask questions up front and you just start doing the work, right? And then you start worrying about the other problems of scalability and da, da, da, da. And then you get to the point of when you actually need money and funding and justification. Why are you doing this, even? And so bring that to the forefront for everybody's mind of figure out why you're doing this. If it, if this is your idea, cause it's cool, well, good on you. But if you want someone to pay you and give you support for it, you need a reason that they can support from a business perspective.

Chris Romeo: 52:46

yeah, that's, that's good advice. Simon Sinek wrote a whole book on it called Start With Why, which is one of my, when people ask me my top book recommendations, and it's an old book at this point, it's probably 10 years or more old, but, uh, it's, it really changed my thinking in regards to program building and really doing anything, starting with why. Just change, it changes your, your way of thinking. Instead of diving into a solution, you step back and go, Why am I doing this? And it's amazing how that changes the scope of where, what you start working on after you have, after you spend some time thinking about it as well. So, Jason, thank you for sharing your threat modeling experience and program building and the three pillars. This has been very educational for me and thought provoking, kind of challenged my thinking in some different areas, which is excellent. So, really enjoyed the conversation and look forward to another conversation in the future on threat modeling or maybe some other topic.

Jason Nelson: 53:41

Oh, thank you, Chris, for having me and, uh, I'm just imagining, uh, you know, Robert riding in the off into the sunset with Rip in Yellowstone right now.