Varun Badhwar -- The Developer Productivity Tax

Varun Badhwar is a three-time founder, a luminary in the cyber security industry, and a clear communicator. He joins Chris and Robert on the Application Security Podcast to discuss scanning with context, SBOM plus VEX, and the developer productivity tax. The concept of a "Developer Productivity Tax" acknowledges the challenges developers face when bombarded with a plethora of vulnerabilities. This "tax" represents the drain on developers' time and resources as they navigate through a myriad of potential threats, many of which lack actionable context. The inefficiencies arising from this process can lead to significant delays in software development, emphasizing the need for more refined tools and techniques.

A key solution Varun offers is the integration of SBOM plus VEX (Software Bill of Materials with Vulnerability Exploitability eXchange). While SBOM offers transparency by detailing all software components and dependencies, it can be overwhelming due to the sheer volume of potential vulnerabilities it flags. VEX, designed as a companion to SBOM, provides the much-needed context, detailing the applicability, reachability, and availability of fixes for vulnerabilities. This combination aims to streamline the vulnerability management process, ensuring that only relevant and critical threats are addressed.

Lastly, the importance of "Scanning with Context" was emphasized. Traditional vulnerability scanning can often result in a multitude of false positives or irrelevant findings due to the lack of context. The podcast delved into the two primary approaches to contextual scanning: static analysis and runtime analysis. While both methods have their merits, the discussion leaned towards static analysis for its scalability and efficiency. The episode concluded by stressing the need for further research and development in vulnerability annotation to specific code functions, ensuring a more precise and actionable vulnerability management process.

Important Links:

• Endor Labs - https://www.endorlabs.com/

Recommended books:

• The Hard Thing About Hard Things by Ben Horowitz
• Software Transparency: Supply Chain Security in an Era of a Software-Driven Society by Chris Hughes, Tony Turner

FOLLOW OUR SOCIAL MEDIA:

➜Twitter: @AppSecPodcast
➜LinkedIn: The Application Security Podcast
➜YouTube: https://www.youtube.com/@ApplicationSecurityPodcast

Thanks for Listening!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Transcript

Chris Romeo: 0:00

Varun Bhadwar is a three time founder and a true luminary in the cybersecurity landscape. Currently at the helm of Endor Labs, a startup dedicated to software supply chain security, Varun has an impressive track record, including building Palo Alto Network's cloud native security business from the ground up and founding RedLock, which was acquired for a whopping 230 million. We'll dive deep into topics like the relevance of CVSS in today's AppSec world, the intriguing concept of VEX and its connection with SBOM, and the ever evolving landscape of developer productivity. We'll also explore if the shift left movement has pushed the boundaries just a bit too far. We hope you enjoy this conversation with Varun Bhadwar. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I am the CEO of Kerr Ventures and CEO of a stealth startup in the cybersecurity space. I'm also joined by Robert Hurlbut, who is broadcasting from apparently the largest room he could find.

Robert Hurlbut: 1:52

That's right, yes. Hey, Robert Hurlbutt, I am a principal application security architect and threat modeling lead at Aquia, and yeah, just been traveling and found a good place to to broadcast today, and you know, that's how it is, right? Sometimes you just have to figure out what works, and go with it.

Chris Romeo: 2:10

You got to broadcast from wherever you are. So Robert is in a ballroom of a hotel where you could have a good sized wedding if you knew 500 people. or more. It would be a perfect place. So, let's jump in. Let's meet Varun, who is our guest today. And Varun, we always like to throw people right into the security origin story. So, tell us about how you got into application security. What was your journey like?

Varun Badhwar: 2:36

Sure. So Chris and Robert, thanks for having me. You know, my, I've got a bit of a funny story. So I did my undergrad in computer science and I just was a horrible software engineer, like didn't see myself writing code as my day job. And, but I love being kind of in the technology realm. And, and in that kind of, you know, soul searching phase, I happened to take an elective course in ethical hacking. And this was long before there was kind of degrees in cybersecurity available in undergrad programs. And I just love the fact that you could break stuff and not go to jail. And I'm like, hell yeah, this is great. How do I find a job in this category? But, you know, we're talking 2006, people didn't hire folks out of college for cyber security jobs back then. And so I ended up getting two offers for kind of consulting, technology consulting, one from Deloitte, one from KPMG. And I basically played them against each other to say who would allow me into their kind of security consulting practice. And lo and behold, both did. And I ended up joining KPMG, did that for a couple of years, ended up at Salesforce.com when there were 1500 employees. I was the fifth hire in their security team. Um, saw this amazing burst of activity in cloud and. You know, honestly, in 2010, I looked back for the last four years at Salesforce and said, you know, I know more about cloud security at this point than probably 90, 95 percent of the market. And how do I use that to do something interesting, exciting and pivoted to start building companies. So my first company, a company called CipherCloud was the first ever CASB in the market. And, uh, you know, the term CASB was defined three years later. And then did that for about five years and saw this next evolution of the move from data center to the cloud. Ended up, uh, starting a company called RedLock, which was acquired three years later by Palo Alto Networks, uh, became Prisma Cloud as what it's known today. But again, it was a category defining company. Like, if you think about cloud security, posture management, and CNAPP, RedLock, Prisma Cloud were the creators of a large scale of that category. You know, I've been proud of the work we did there, but in that journey at Palo Alto Networks, it kind of, I had 400 engineers on my team and, you know, we were running SCA tools, we had 68,000 alerts, and this is right around the time when, you know, solar winds happened and people were asking a lot of questions around software supply chain. And, you know, I just uncovered tremendous inefficiencies. In the process, like the tools want to check a box and tell you every possible thing that could be wrong. You shove that report and list to your engineers and 8 out of 10 times, you know, those things are just not relevant for the way your applications are being built by your developers. But, you know, what I absolutely hated was it's a guilty until proven innocent model. The tools find the developers guilty. They give you the alerts and ultimately the developers have to come back and fight for exceptions and convince you that they're innocent. Something is very wrong with that model. And that's kind of led me to the founding and genesis of Endor Labs, where, you know, we're working on really eliminating the developer productivity tax. And I'm sure we'll talk about more of that, but that's, in essence, who we are and what we do.

Robert Hurlbut: 5:46

Yeah, very cool. So, let's, let's dive into that. You know, what is developer productivity tax? What does that mean?

Varun Badhwar: 5:55

Yeah, you know, look, when we hire software engineers, like the OKRs we typically have for them is how much code are they shipping, right? One fundamental problem we've had in security is like their OKRs are very different from somebody in security or application security. We'll get to that probably later. I think the fundamental thing around developer productivity tax is it's all of the unnecessary work that developers have to do above and beyond just securing their applications or developing by, you know, secure by design. Thank you. It's the stuff where they have to keep fighting to prove their innocence on lots of alerts from lots of AST tools that kind of make no sense for their particular use case or application. It's also fighting to deploy dozens of different security tools into your pipelines, which are constantly churning and changing. And, you know, you have 300 pipelines. It takes two wheels, two weeks to deploy tools. In certain pipelines, it's added work that is honestly not something that they are getting measured on because their teams are saying how fast are you building software, how fast are you shipping releases, and how good is the quality, right? Like, ultimately, that's what most companies are measured on. And the third thing is this whole exception handling process, right? Like, you know, Snyk will say you have 10, 000 vulnerabilities. Okay, now the developer will take eight hours per vulnerability to go review the report and determine what is relevant, what is not. You know, CVSS prioritization sucks. We'll talk about why. Um, it's fighting all of that bureaucracy in the process to just convince people that they should still keep doing what they're doing. That is the developer productivity tax. And where we see that tax being the highest today in the application security realm, first and foremost, if we acknowledge and agree that 80 90 percent of code in modern applications is code you didn't write, but you borrowed from complete strangers in the open source ecosystem, that That is where a lot of your alert fatigue is coming in from, is SCA and open source governance in general. The second thing where the fatigue and alerts are coming in is secrets. Like if you think about in the pipeline, we all want to put a secret detection tool. I mean, those tools could get really noisy, really, really fast. You know, what secret's valid? What is something that is historically there, but is not even a valid secret anymore? I've already kind of disabled it. How do I know who's tracking it? I got to keep parsing through these reports. And the third thing is the tax of just running all of these tools in the pipeline that are disconnected, not communicating with each other. And, and I have to constantly manage these because my security engineer wants me to, you know, put a new tool and then they want to POC a new tool, but all of that work keeps coming on me in DevOps to run those tools.

Chris Romeo: 8:39

So, let's talk about CVSS. Because I know you've written a lot about CVSS, and so this might be a bit of a loaded question, but I'm going to ask it anyway. What's the value of CVSS in the modern world of application security?

Varun Badhwar: 8:55

Look, it is one of many indicators that you probably should use to prioritize what you want to go work on next. The way I simplify the life of vulnerability management programs is there needs to be a now, next, and never list. So if we agree that there needs to be some kind of a model, you can call it whatever you like, but a now, next, never in my mind, the question is what is in the now list? I think the mistake a lot of organizations make is they will go look at alerts and outputs for any tool. You know, let's, I already mentioned Snyk, let's just continue picking on Snyk. You know, you'll take a Snyk report, you'll say criticals and highs, filter. Okay, I'm down from 300, 000 alerts to 100, 000 alerts. Okay, here goes Jira tickets for 100, 000 things to my engineers. It doesn't make sense. First of all, you know, that is one segment and approach to looking at prioritization, but it's missing key factors. It has no context of exploit maturity. Is there even an actual exploit for this or is this a theoretical thing that some security researcher found that, you know, there's, there's no real active attacks on? So we think things like EPSS, right? Exploit Predictability Scoring System that is now getting a lot of industry standardization and recognition. As a vendor neutral mechanism to look at exploit maturity is another important indicator to add on. You know, beyond that, we think the biggest thing, the most relevant thing to your organization is your internal applicability of that vulnerability. What do I mean by that? 90 percent of your code is open source? Fine. What our research has found, only 12 percent of that 90 percent code is actually used in your application. So, simple example, I'm using an AWS SDK. The AWS SDK supports 100 AWS services. My app is using EC2. If I'm only calling the function that connects me to EC2, and that is vulnerability free, why do I have to go drop everything in fixed vulnerabilities and the lambda function calls or the S3 calls that are part of that SDK, but that code is not even getting invoked in my application? And the fundamental knowledge around reachability, am I actually using the part of the code that has a vulnerability? Is it internet facing, right? All of these factors, is it a test dependency or is it in production, are super important and we think you need a well rounded aggregate mechanism to look at prioritization and risk. And by doing so, you can cut your list down by 90 percent and really focus the now on drop everything. Here are the critical with exploits that are mature, reachable in my application. Let's go fix those handful now and let's get to everything else in my just regular release process. And not disrupt the developer flow. That's kind of how we perceive, you know, how prioritization should occur. Uh, not on a single metric.

Chris Romeo: 11:50

Whenever I, whenever I hear about reachability, I'm always curious to understand how, how do you truly measure reachability? Like, without being inside of the running application and having some type of observability layer that can see if something, like, how do you measure? Like, when you think about reachability, how is this thing measured? How do we know if something really is truly reachable?

Varun Badhwar: 12:15

Yeah, look, broadly speaking, there's two approaches here. And by the way, I completely agree. You've had, um, you know, speakers in the past come and talk about it. And Jim's talked about how, you know, SCAA as it is today with scanning manifest files is dead. It really needs static analysis approaches and SAST techniques to do it. So one way is you do it statically. You do static analysis, specifically to trace how your first party code is calling these dependencies, how those dependencies are calling those transitive dependencies. Which functions are you calling? Which ones are you not calling? It's a very hard computer science problem to do, but it can get you 80 percent of the way there, okay? That's one way to do it statically, building a control flow graph. The other approach is doing it through runtime agents and kind of something dynamic in runtime. That is much more intrusive as we know every time trying to convince people to put something in security in runtime is hard. So people will kind of fall back and say, well, let's run it in test to run it in staging. The problem there with these runtime approaches is not only are they more intrusive, but they're also prone to false negatives because it's only as good as the mercy of your test code coverage. If the test codes are executing against 30 percent of your application, well, the only scenarios that are getting played out in your staging environment is to see If any of those libraries are getting loaded in that 30 percent of the calling application. So, stepping back, you can do it statically, you can do it through runtime, each one has its pros and cons. We fundamentally at Endor Labs made the bet that static techniques are going to be more efficient and scalable in the long run. Nobody wants another agent, I don't want to fight that battle with my SRE teams. I'm trying to reduce the tax, not add a tax. And so the way we do it is statically, we've built it for several libraries and programming languages. So we can do it for Java, Go, Python, C#, Rust, and others. But the technique is basically looking to say, are you calling, which functions are you calling in your application? So that's one part of the problem. And how do you do it all the way down the transitive tree of your dependencies? There's a second part to this problem. No matter which approach you use, none of the existing CVE databases have the vulnerabilities annotated to the function. So you have a whole security research component that needs to go annotate these vulnerabilities to say, well, what part of that open source library, what function, what line of code was a vulnerability in? So then when I scan your code, I know what code, what lines you're using. I know where the vulnerability is, I piece it together, and I ship the results to you. Um, so it kind of is a two party problem, if you will.

Robert Hurlbut: 15:00

Okay. So, here's another question for you. Uh, we've heard this term VEX, but, um, both Chris and I were, we don't really know a lot about it. We want to learn more about it. Uh, so what is VEX, and how does it play with SBOMs?

Varun Badhwar: 15:17

Yeah, so let's kind of pick up where we were, right? We talked about importance of context. Context of application, context of reachability, context of, is it actually, uh, a test dependency? Is it in production? It's context of, um, is there a fix even available? Like, great, you're telling me there's a vulnerability I should fix, but if the maintainer hasn't even shipped the upgrade, it's not. So there's a huge amount of context that's important to understand about set of vulnerabilities. So now let's tie this to SBOMs and then get to VEX. Well, what is SBOMs? To me, an SBOM is a means to an end. It's, we all want transparency in software. Makes a lot of sense. We all want to understand what are the ingredients of our software. Great. So an SBOM is basically writing, we're using this package, this version, this package, this version. Now, what do you expect somebody who gets an SBOM to do, right? So Chris sends me an SBOM, I am your consumer of Chris's application, I pick my favorite tool, and I run Chris's SBOM against it, right? Let's say I take OWASP dependency track, I import it, that is going to tell me, hey, Chris's application has these 10, 000 vulnerabilities. I have no context of Chris's application, I just know based on the version and the name of the packages. Here are the 10, 000 problems he has. I'm going to go back to Chris by email, send him a 10, 000 line entries in a spreadsheet and say, Chris, why aren't you fix these? Chris will go to the developer and say, Development team, my customer is asking me all this stuff. Why didn't we fix this? Well, this is not a scalable, sustainable process. So, as an industry, one thing is we talk a lot about SBOMs, but let's be pragmatic about what are we going to do about SBOMs. If we do what I just described, which is what a lot of the industry is doing, this is dead on arrival. Chris doesn't have the resources to scale. I don't have the resources to chase Chris down. Dead on arrival, we check a box and we call it a day. The way to make this much more practical is that the same body, NTIA, that created the SBOM standard created a VEX, a Vulnerability Exploitability eXchange Format. And they describe VEX as a companion document to SBOMs. And I call it, it's like, SBOM without a VEX is like peanut butter without jelly. Because if I'm, if Chris is sending an SBOM to me and I don't have the context of preemptively Chris telling me, look, your scanners might tell you there's 10, 000 things. Here's reasons descriptively in a JSON format. I've chosen not to fix this because this is not applicable to me. This particular thing is not reachable. This thing doesn't have a fix available. This thing is a test dependency. I don't use it in production. That's the kind of context that the VEX carries. So in VEX you can mark things like not effective. Not used, right? Those types of descriptors. And the idea is that if you send the SBOM with the VEX, I have enough context not to have to come back to Chris with 10, 000 things. Maybe I'll come back with five things. Maybe I just won't come back. But this is how you make this much more pragmatic and programmatic to be able to communicate risk along with all of the exceptions and the reason for exception because I'm sure we will all agree, we will never ship vulnerability free software as somebody may define in its kind of true sense. What you might send, be able to ship is software that is Vulnerability free from vulnerabilities that actually affect you and can affect your downstream customers. And that's kind of the goal of SBOM plus VEX is being able to communicate that. Now, the problem in the industry though that we lack is 100 tools to generate SBOMs. Really no tools to generate VEX automatically. So that's been a whole other area of research and development and focus for us, which, you know, people can read more about on the Endor Labs website.

Chris Romeo: 19:13

Okay, so the perfect state in the future will be a tool that generates the VEX file automatically, to determine....Now, what is it, what, what, what's, what are the rules? What are the questions that that tool has to ask to be able to determine if something is truly one of the five things I should care about or not?

Varun Badhwar: 19:39

So, think of it this way. Today, how do you manage exceptions internally, right? Internally between development teams and security teams, you probably have an exception register or JIRA tickets where you say, security says fix this, engineering says I will not fix it and here's the reason, and you say, okay, fine. It's automating that and annotating that. That's the kind of information you put in a VEX. So VEX has a template, it has a format, it has reasons. Right? Not affected, not reachable, bunch of stuff. And so with the CVEs, you can proactively say, Look, this CVE, it's here in this library version, and here's the reason we're not fixing it. Now, can you automate 100 percent of it? Probably not. Can you automate 70, 80 percent of it? Yes. Right? Like, I already know if it's a test dependency or it's shipped in production. I already know if it's reachable or not reachable. I already know if... The whole package is still being used or not in my application. Um, I already know if there's a fix available or not available. Now the things I might not know is the inputs to the application, right? Because some of these vulnerabilities are dependent on various inputs and other techniques. Or do you have any compensating controls in front of your application? That I might not be able to discover automatically, but your developers might know and security teams might know, so you can further annotate those. But if you can even generate 70, 80, 90 percent of the VEX automatically, that is a huge win, uh, for you.

Chris Romeo: 21:06

I mean, I think it's gotta be a hundred percent to be. Truly beneficial because in a DevOps world, right? And this is, this is one of the problems I've had with SBOM. Okay. So I'm, I'm, I've become known as somebody who's not drinking the SBOM, whatever, like not, not getting excited about SBOM. Like, I'm just, I'm not that excited about it. But I, and I believe in SBOM from the transparency where you, where you kind of started this. I think that's, I think that's crucial. I want transparency, but everybody's gotten so excited about SBOM. Like you said, there's hundreds of tools for generating the SBOM formats, but the challenge is. I don't think people are going to use them. And I don't think people are going to get the value out of them based on the effort that is going into, to creating them. And so I look across the industry and everybody's SBOM positive and like, Hey, we love SBOM, SBOM is the greatest thing. Like, no, SBOM is not going to solve all of our AppSec problems. I'm sorry, it's not going to, and it will provide transparency as long as an organization can find a way to consume it in such a way that it adds value. And you mentioned OWASP dependency track, that's an example of a way you can, you can consume SBOMs and, and, and derive value from a collection of SBOMs. I get that, and, but I don't know, I mean, any thoughts based on my naysayer SBOM opinions?

Varun Badhwar: 22:33

Several thoughts. One, you're right. We talk to a lot of customers that are asking for SBOMs and we say, what do you do with them? Well, today the answer is I file them under a desk or file them under a virtual SharePoint desk and call it a day. They recognize that's a problem and... You know, they're going to get better at that. But here's my fundamental issue, and why I call an SBOM a means to an end. By the time you generate an SBOM, like think about this, how much do you control the inputs of what went into your application? You're just now presenting the components in your application, but as a, as a application security leader, how much did you control what came into this pipeline at the front gate of your house, right? You're just saying this is what's going out. Great, but at this point, it's a wall of shame. What we are really after is shifting that whole thing all the way left to the point in time when the developer says, I want to use a new open source project. Okay, great. What are you hoping to do? Well, I need a logging library in Go. Great, let me help you. Let me recommend to you what you should use that I know will get a clean bill of health later. Right, like that's the early intervention. Or, you know, here, you have three choices between package A, package B, package C. Here's all the metrics on them. Here's like your, you know, uh, here's the human risk factors. Here are the people behind these projects. Here's how well the code is written. Here's it's vulnerability free today. It has good documentation, good release practices, well supported. You should probably use it and you're likely to be okay. Like, you need that early system in Gates because that is when you control and can influence what goes into your application because otherwise you're just fighting the fire of applications getting shipped. Here's an SBOM. Holy shit, that SBOM is showing me that I have a package that is 36 releases old. Why didn't we do something about it? Well, guess what? If I was reviewing the PRs. When your developers were pushing this code, I could intervene very early, give them that input, and I wouldn't have this problem. So, to me, SBOMs without this whole, what I call an open source governance program, is, to your point, basically a check the box, um, exercise. My hope is people will use these wall of shame reports initially that'll come out, where people will get asked a lot of questions eventually when these come out of a file cabinet and get put in some system. And that will force us all to have better governance and guardrails there where the answer is not to not use open source. That would be a horrible outcome. Open source is generally good for humanity, but do it responsibly and do it confidently in a data driven approach.

Chris Romeo: 25:09

Yeah, our, uh, our friend Izar Tarandach has made a shirt that says"my threat model is better than your SBOM." Exactly, like it's because to your point like when I think about SBOMs, I mean there are companies in the SaaS world that push 500 releases a day. If you don't have an automated high speed way to process those as a consumer, you can't do it. If someone's filing 500 of these things under a desk, In your virtual desk kind of example, like that's, that's zero value and for big companies to generate, create SBOM programs where they are ha, you know, cre releasing, building these things into their products. You think about big product companies with thousands of products. This is real money. It's real dollars associated with this. And so if we're not going to get the value out of it If I'm a big company, I'm saying, I'm not doing it. You, you prove to me that there's a value proposition. And I know what's happening is customers are saying, well, we're not going to buy if you don't have an SBOM. And companies are going to bend because they want to sell product at the end of the day, and they want to make money and all the other good stuff that comes from, from running a company. But if you're not getting, if you're not getting benefit out of it, that's, that's where I've, that's where I've been stuck. It's like, show me a way where we're getting the right amount of, of return on investment on my dollars. And then I'll, I'll, I'll, I'll become a, an SBOM, I'll wear a shirt that says SBOM. But right now I'm wearing, I'm wearing, I'm going to wear a Izar shirt that says my threat model is better than your SBOM.

Varun Badhwar: 26:44

Look, I think, I think starting checkbox, it has to shift to real security. Um, we, we actually wrote a blog post, uh, people can check it out on our website. It's called SBOM versus SBOM, because we ran three of the most popular open source SBOM generation tools, and all three against the same application gave different results. So the big question is like, what do I believe, right? So there's a whole attestation problem, accuracy problem. Lots of stuff has to be vetted out. My opinion, the trains left the station. Like yes, SBOM is now becoming a compliance necessity, but hopefully we'll take it much beyond that and not end it as being yet another compliance checkbox that doesn't really mean anything.

Chris Romeo: 27:25

Yeah, yeah, I've said some of those same words as well. That's my, that was another conclusion that I had as well is like compliance is just not the way for true security. It never has been. And unfortunately, it's a minimum standard. That's been applied. And we get some small security improvements out of compliance, but we don't, we don't get the, we definitely don't get a return on investment from a business perspective where you can look at that and say, I invested this much money and it resulted in me. Getting this much gain for us as a company. It's not there in the, any, anything in compliance doesn't have a solid return on investment. It's more of a loss leader type of thing where I have to, I have to do these things in order to sell. And so maybe you could say there was a return on investment, probably be zero actually if, if I'm spending these things to just to try to sell my product. Yeah. You

Varun Badhwar: 28:17

Yeah. Look, I think this is why Chris and Robert, we started with developer productivity tax. Our mission in the world starts with that, uh, because we think if you, if you really drive the right outcomes where, uh, you are minimizing that developer productivity tax and along that way, checking those compliance boxes along that way. You know, improving the posture of risk, improving the resiliency of the applications. That's the focus, but I think today, starting a product or a company, just focus on compliance, you know, I'm not sure. I wouldn't personally do it. I think we're focused on much bigger issues in application security and developer productivity.

Chris Romeo: 28:52

Yeah, you might as well have a result. You might, you might as well do something that could change the world. Like, who, nobody starts a company, I mean, some people do start a company and say, I just want to build compliance into things. I don't know, that's, for me, I'm not going to get out of bed and be super excited in the morning. What are we doing? We're going to make another, another, another clipboard with a bunch of things on it. We got to check off. That's what we're doing. Like, I don't know. I mean, you've built a number of companies. It's, it's, there's something about building a company. You want to do, you want to save the world. You want to, you want to do something of, of meaning. Like, you don't want to create a company that's just here to make more check boxes. That's

Robert Hurlbut: 29:30

So are there any other tools or things that AppSec teams could use or should use?

Varun Badhwar: 29:39

Oh boy. I think the problem is there are too many tools that the AppSec teams can use in the questions around consolidation. I think my thesis is the whole shift left movement has gone a little too far in principle. 100 percent makes sense, right? Like we've got to do things early. We've got to intervene early. Full believer in that. But the way we're doing that today is just so inefficient. Like, you know, you have one tool you run in your pipeline for secrets, one for SAST, one for SCA. You know, another thing for IAC, another thing for container image scanning. And then your, your platform teams are so frustrated just managing these tools. And they don't have dedicated headcount to just run tooling for you and instrumentation for you. And so I think, I think that the, the right approach architecturally is to have this concept of a parallel security pipeline, which is basically to say, look, you instrument this one thing. This gives us the ability to hook in. 30 tools if we want to, or put in one best of breed tool if we want to, however you want to do it. But you kind of abstract away the need for a platform engineer to have to go intervene every single time, every version update, every breaking change. And you kind of abstract that down to say, okay, as an SRE organization, you give me this. This hook into your pipeline, you tell me how much time I have, how much CPU, how much memory I have, and then let me as a security organization orchestrate the tools myself, independent of you, so I don't come bother you every single time, but we agree upon the constraints and confines, and you want to run 30 tools, have at it, you want to run one tool, that's your choice, you want to change tool every month, you go do it as a security organization, but, uh, you know, we kind of bring a little degree of separation and paralyzation in the pipeline.

Chris Romeo: 31:21

Do you see a consolidation coming in the market? I hadn't really thought about this until you were describing it in such a way of 18 different tools, but then you start, my first reaction was, oh, there's not that many, but then you start to list off the things like, okay, uh, secrets management, infrastructures codes, SAST, um, IAST, uh, you keep going down the list. There's more container scanning.

Varun Badhwar: 31:47

Yeah.

Chris Romeo: 31:47

Is there a consolidation coming in our market?

Varun Badhwar: 31:51

Look, I think consolidation has always been an interesting point. If you look at what we did with Prisma Cloud is, you know, we had a fundamental belief that customers will choose best of breed products. But what if you could integrate best of breed products into an integrated platform? So can you give both? Now, you know, with the, with the resources at Palo Alto Networks and the several, um, acquisitions we made there, we did manage to deliver that. We think for a single company to try to do everything is nearly impossible. So then that means... How do you deliver similar outcomes but give customers the flexibility to still apply best of breed tools? So I think where the opportunity lies is just like what Docker and containerization did to the world of compute, you could do on orchestrating security tools. This kind of goes back to my concept of parallel security pipelines. If we could give a container in your pipeline that could run whatever security tools you want to do, you simplify the deployment to one thing in the pipeline. So if you deploy this one hook, you can do whatever you want. You know, there's also Gartner's whole new category of application security posture management. output side, right? So once I have all of the alerts, how do I aggregate them? So the AppSec teams and developers don't have to go to 25 tools to review results. How do you de-duplicate? I think that also makes sense. So the, the sensors, if in effect, the tools might be different, but they could get connected to one thing, right? Ultimately, and then just kind of plug and play. And then you have this AFBM concept that then brings all of the results from those tools back into a singular Um, results set deduplicated for the user, that makes a lot of sense too. So I think it'll be, you, you be vendor agnostic if you want to be and run whatever tools, you know, year one, you want to run tool A for SAST, year two, you want to switch it out. Complete flexibility with smart orchestration.

Chris Romeo: 33:45

All right, Robert, it's, uh, it's time for the lightning round. So why don't you take us into the lightning round with Varun.

Robert Hurlbut: 33:52

Okay. Uh, yeah. So we have a few questions that we typically ask. So first of all, what's your most controversial opinion on application security and why do you hold this view?

Varun Badhwar: 34:06

Um, I think, I think similar to maybe some other people's beliefs. I think, um, the software composition analysis market and the approach that we know, as we know of it today, scanning manifest files, is dead. We think you need to have a much smarter approach, uh, that is much more targeted to really understand issues related to the parts of the code that you use. That's one. We also fundamentally believe that, uh, or I personally fundamentally believe that just focusing on license risks and vulnerability risks as software composition tools have done is not enough. The whole supply chain and software supply chain risks. Has a lot of unknown unknowns, malware in packages, name confusion attacks, poorly maintained software, provenance issues. These are things your SCA tools that focus on license and vulnerabilities just don't catch. So I think, uh, you know, uh, and, and you're probably going to ask me another question related to this, but I think there's a better way than SCA. And it's kind of like the shift we saw in the endpoint market where we moved from signature based AV products to eventually now, you know, EDR and XDR tools. We need to see that shift now around a lot of AST tools, especially around SCA.

Robert Hurlbut: 35:21

All right. Next question. Uh, what would it say if you could display a single message on a billboard at RSA or a Black Hat conference?

Varun Badhwar: 35:33

There's a better way than SCA to manage your open source risks.

Robert Hurlbut: 35:38

Makes sense. All right. Last question is, what's your top recommendation for, um, a book? Uh, it could be security, it could be something else. Uh, and why do you find it valuable?

Varun Badhwar: 35:52

Yeah, I'll give you two, um, two things. One is a lot of people listening in here are probably security professionals who may, you know, like me, want to move into entrepreneurship. They have great ideas. You know, I find a book The Hard Thing About Hard Things by Ben Horowitz. It's a great book if you kind of want to go down the entrepreneurship path and want to understand what the journey looks like. I think relevant to application security and software supply chain security that we just discussed. There's a new book by Chris Hughes and Tony Turner around software transparency and supply chain security in the era of software driven society. I think that's a great read for people wanting to come up to speed with like, what the heck is this software supply chain security thing and why is everybody in the world fixated on this over the last two years? It's a great, it's a great read.

Chris Romeo: 36:40

Cool. Yeah, Tony Turner's a past guest. Chris Hughes is a future guest. So, I love that, that that recommendation came full circle. Now I gotta read the book, like, now I have many reasons to read the book. Alright, Varun, so from a key takeaway perspective... To finalize our conversation here, we talked about a lot of things from developer productivity tax, to CVSS, to SBOMS, to VEX, to, you know, what you would put on a billboard at the RSA conference, but what, um, you know, kind of from a key takeaway perspective, what do you want to, what do you want to leave our audience with?

Varun Badhwar: 37:18

Yeah, I think, um, I think it goes back to what we discussed, right? If the audience, you know, a lot of listeners are application security engineers. One, you know, we have this whole concept of lean AppSec. Like what we find is typically it's like one AppSec engineer to several hundred engineers. So how do you scale yourself? But also how do you have empathy that the development teams don't have time to just chase your alerts and review code and then discuss and debate with you. So how do you bring in a lot of efficiencies into this process is a must today in our opinion. And I think kind of more, you know, uh, kind of refreshed approaches around things like SCA that we discussed and more simplification of the pipelines. I think is going to be critical to scaling any application security program moving forward. So, you know, feel free to ping me on LinkedIn if you're interested in chatting more about this subject. I have lots of thoughts and opinions on this. Or, you know, check out a lot of great technical content on the Endor Labs website. We also recently did a Lean AppSec conference, virtual conference. You had great speakers and practitioners. If people want to know more about the concepts I've discussed, you know, feel free to take a look at that content that's available on demand.

Chris Romeo: 38:30

All right. Thanks Varun for being a guest, for walking us through and teaching us about VEX, as Robert and I have admitted, we didn't really know a lot about it. So it's helpful to, to get that perspective. Uh, but thanks for being a guest on the Application Security Podcast. And, uh, we look forward to a future conversation on some other topic.

Varun Badhwar: 38:48

Yeah, thanks so much for having me and have a great trip, Robert.

Robert Hurlbut: 38:52

Thank you.