Mark Curphey and John Viega -- Chalk

Mark Curphey and John Viega join Chris and Robert to explain the details of Chalk, Crash Override's new tool. Mark also talks about why ZAP departed from OWASP and joined the Software Security Project, highlighting some of the value and differences of both organizations. Open Source Software is important to the industry, but Mark calls on companies to contribute to the development and support of the projects they use.

The conversation explores the challenges faced by companies, especially large tech firms, in managing their software engineering processes. Many organizations grapple with identifying code ownership, determining code versions during incidents, and prioritizing alerts from static analysis tools. Chalk emerges as a solution to these challenges, providing clarity and reducing friction in the software development and maintenance process.

Toward the end, both speakers emphasize the importance of understanding the entire software engineering process to make informed decisions. They advocate for an "outside-in" perspective, urging listeners to step into the shoes of others and view challenges from a broader perspective. This holistic approach, they suggest, can lead to more effective decision-making in the realm of software development.

Listen until the end for book recommendations on cybersecurity, business, and personal growth.

Links:

• Crash Override: https://crashoverride.com/about/
• Chalk: https://crashoverride.com/docs/chalk/overview/
• The Software Security Project: https://softwaresecurityproject.org/
• The Open Worldwide Application Security Project (OWASP): https://owasp.org/

Books:

• Cybersecurity Myths and Misconceptions... by Eugene H. Spafford, Leigh Metcalf, and Josiah Dykstra: https://www.pearson.com/en-us/subject-catalog/p/cybersecurity-myths-and-misconceptions-avoiding-the-hazards-and-pitfalls-that-derail/P200000007269/9780137929238
• Crossing the Chasm by Geoffrey A. Moore: https://www.harpercollins.com/products/crossing-the-chasm-3rd-edition-geoffrey-a-moore?variant=32130444066850
• The Pragmatic Framework: https://www.pragmaticinstitute.com/product/framework/
• Atomic Habits by James Clear: https://jamesclear.com/atomic-habits
• Start with Why by Simon Sinek: https://simonsinek.com/books/start-with-why/

FOLLOW OUR SOCIAL MEDIA:

➜Twitter: @AppSecPodcast
➜LinkedIn: The Application Security Podcast
➜YouTube: https://www.youtube.com/@ApplicationSecurityPodcast

Thanks for Listening!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Transcript

Chris Romeo: 0:00

John Viega is the CEO of Crash override, and Mark Curphey is the CMO. John and Mark have a long history in application security from authoring the first ever book on the topic, founding OWASP in 2002, and the Software Security Project in 2023. They recently founded two successful application security companies, SourceClear and Capsule8. John and Mark join us to talk about Chalk, the problems it solves and what Chalk marks provide for developers, and also to share thoughts on the ZAP move to the Software Security Project. We hope you enjoy this conversation with John and Mark. Hey folks. Welcome to another episode of the Application Security podcast. This is Chris Romeo. I'm the CEO of Kerr Ventures, and joined by my good friend Robert Hurlbut. Hey, Robert. I.

Robert Hurlbut: 1:33

Hey, Chris. Yeah, Robert Hurlbut. Um, I'm a principal application security architect and threat modeling lead over at Aquia. And really good to be here today with you and our guests.

Chris Romeo: 1:45

Yeah, this is, uh, gonna be a fun conversation. We have, uh, Mark Curphey, who's been with us before. I talked a lot about, uh, the history of OWASP and things that, uh, kind of filled us in on a lot of that. But we're, we're gonna talk about something different today, um, at least to start as far as the open source project that they've been working towards. But first, let's jump in and introduce John Viega. Um, John, we always like to hear people's security origin story as a way to kind of just jump into the deep end really fast. So how'd you get into this world of application security?

John Viega: 2:17

Totaly on accident. So I was in grad school doing compilers in programming languages. I decided I needed to go to the real world, worked, uh, got a job at, uh, a, a company that at the time was called RST, working for a guy named Gary McGraw. And, uh, ended up doing a bunch of that was all security work, so, uh, took my compiler knowledge to build like the first static analysis tools for security. And, uh, including like the first open source ones and like also wrote the first book with Gary for developers on security. Did. A bunch of books after that, uh, but also was doing a lot of cryptography. So I did the, uh, the, uh, GCM uh, encryption mode that, uh, is the default for TLS and is basically in hardware everywhere. Um, And then I put on a suit and sold out for a while, I guess. And, uh, but, but, but I guess, so I met Mark, uh, when he was founding OWASP because, uh, the book Building Secure Software had come out like less than a year before, and he reached out uh, when he was founding OWASP. So I was, uh, like joined the original tech advisory board with we Sopel and Hoagland and so on. Um, and know that that didn't, I didn't stay around for very long, but, uh, have stayed very close to Mark throughout the years. And, uh, even as I kind of left AppSec, uh, I stayed close with Mark.

Chris Romeo: 4:19

cool. so I want to, I want to kind of dive in. In to our conversation here with a philosophical statement.'cause as I was looking at what you guys are doing now, and I know we're gonna unpack that quite a bit more, I, I found this interesting statement that you put out, and I wanna, I wanna understand this better. And it was this idea that devs can't see into production. SREs can't see into development or production and security can't see either. And so I'm looking at that and I'm like, Isn't that why DevOps was created originally? Like we are, we are there silos that exist that people just have lost track of or think don't even exist? And so I'd love to get either one of you just jump in on this one and, and give us some thoughts about what, what is the state of siloing in development?

John Viega: 5:13

Yeah, it's, it's not necessarily that, uh, they're intentional silos, but they're defacto silos the bigger companies get. Um, the, what we, so basically the origin story for this, uh, project is Mark and I were both, both, you know, kind of semi-retired and wondering like, let's solve some problem that people have. We went and interviewed a ton of people to understand what problems they do have and the, you know, the, this is like one of the first lowest common denominator we saw, like a lot of people getting hung up on. Information they needed that they didn't know how to find or didn't know the people who owned it. And so even in big tech companies that have great SREs, we'd still have CISOs saying, oh yeah, well, or, or not even just CISOs, but you know, people throughout the org saying, yeah, there was an incident, uh, in production yesterday, and it took us like two hours to figure out who owned the code and which version of the code

Chris Romeo: 6:39

Hmm.

John Viega: 6:40

we were looking at. So like, in really big companies you might, uh, never be able to get the answer those people have left, you know, are not sure which build it is. Like all, all sorts of things that, uh, really provide friction and uh, like just from a security perspective works the other way too. Like you might wanna say, okay, well I've done all this. There, there, there's my static analysis tool has a new type of vulnerability that it is checking for. Wow, its giving me tons of alerts across like hundreds of repos. Now which ones are important? Like what, what actually gets used here? Which ones are hackathon projects? Uh, I don't, I dunno. Um, so those were the kinds of problems that we heard, and then we started asking people how they tried to address it and understand what the failures were, and kind of get deeper into like how they needed to operationalize things. And, uh, that that's what kind of led us to, uh, the problem, I guess. So.

Mark Curphey: 7:52

Yeah, I mean, another way, another way of kind of saying, saying exactly the same thing is that when we were doing these interviews with people, kind of, we just say like, what's your biggest AppSec problem? And people basically said, I don't know what to work on now, next or never. Like, I don't know what's the most important thing to, to work on. We've got a limited amount of resources. I've got a limited amount of team. Even if I had an unlimited budget, I can't hire all the talent because it doesn't exist at scale. I'm flooded by alerts,'cause all the tools are generating alerts and somehow we've gotta figure out, it's not about deduping volumes, it's about what's the most important thing. So when you drill into that, you'd say, well, okay, um, you know, how do you determine that? And it was literally like someone who goes around with a clipboard and knows that this application's facing the internet and its managing PIR. Or, you get these stories like that. So ultimately, what that led us to, to, to figure out is if you wanna go solve that problem, you have to understand the end-to-end software engineering process and all the big pieces about it, and then you can make an informed decision. So, you know, the way, the way I like to describe Chalk is basically it's telemetry and observability for the security engineering process, right? So you can, you know, collect the tel, put the telemetry in for everything from the, when the developer creates a code to when it's built, to when it's deployed, and then you can go and observe what's happening. So, as an example, you know we've got a gaming company that are an early design partner. They've got 6,300 repos. Dependabot runs in every single one. They don't know which one's are hackathon projects, but I can tell you that all 6,300 are not deployed. So, you know, how do I go focus on the, on the hundred that are deployed? Because first thing I've gotta know is which hundred are deployed and then within those hundred, like which ones are managing PII, which ones, which ones aren't? And then when I figure out I've got a V and the one that is managing PII in production, who owns the code for it, right? Who do I, who do I go talk to? But even things like, I've now found a vuln in this app. That was scaned in production by a DAST tool or, or something. You know, what actually was the commit that was built and is pushed on that because it may actually be the wrong one in the first place. It may be one that's been fixed, or hey, you know, the developer said there's a fix here. Like the actual fix has not been deployed yet, so, What we, what we realized is that you can have a massive impact instead of like trying to figure out to vulns, like get rid of 90% of them in the first place cause it matter, cause these repos that aren't being used and are important and things like that, right?

John Viega: 10:19

Yeah. It was really like, we watched people's decision making processes and what info they didn't have. And what could be done to kind of automate that without, uh, friction. And so like a, a a lot of it was connecting the dots between data silos and, uh, you know, the, the like. Again, for a developer that is important too, to Mark's point, like, okay, let's say that there was a big production issue, and you wanna know which version of the software that was actually deployed over there, which the developer often doesn't know. Uh, and even if you do, if you're that removed from the operational environment and you're trying to reproduce something and people don't want you in production, sure would be nice if you had, uh, telemetry. About what the environment was like that it was running in, and what it was talking to, and like, what else was on the node that might actually impact the bug you're seeing. And, uh, none of that stuff is, uh, uh, gonna easy to get either. So, uh, that, uh, like was a queer huge problem. And then tackling it in a way that was gonna be seamless, like as frictionless as possible, uh, for people like easy to just kind of stick it in and forget about it until you need it. Uh, that that's really what, uh, kind of the motivation has been because, you know, we looked at, uh, listen to a lot of people tell their stories about what they tried and we learned that like pretty much any friction is enough to derail it.

Chris Romeo: 12:12

Yeah, so the, so certainly the, the, what you've just laid out is an enterprise class problem. But when I think about this, what you're describing as far as the problem space, I think it still exists in even small companies and even startups. That having that visibility into what code is where and what code's produced. So when you, when you think about this problem space and, and maybe you heard from some people as well that you interviewed, is, is this something that smaller companies and startups suffer from and they could benefit from the same, the same approach.

John Viega: 12:50

Yeah. 100 percent. Like we, we've, uh, For instance, around spend management, uh, we've heard a lot of startups say, well, like anybody can spend anything up anywhere. And, uh, we, like, we're probably spending on a bunch of stuff that we don't know that we're spending on and don't have the visibility. So, um, or like they want, uh, to, uh, to get, uh, more telemetry from the app in production and don't have a good way to do that without, uh, kind of totally re-architecting it. And those are all like, uh, things that we're doing or looking to do, uh, as Chalk evolves. Uh, for instance, like in the serverless world. That's a very constrained environment, and doing any kind of debugging in a production world is not that fun, is what we've heard from a lot of people. And, uh, uh, you know, we're, we're gonna be doing everything we can to make it easier, a lot easier to get, uh, kind of that info out. And then on the other end of the scale with the enterprises, You're right. I mean it's definitely the, a lot of these problems are enterprise problems, but we were really shocked as we started like bringing around an early prototype how like even the biggest companies that talk about it as if they've got the problem licked, don't. So for instance, financial institutions, you know, they've got regulatory requirements to track all this stuff. And so when we were talking to them, we'd say, oh, this isn't, but it was kind of the opposite. Well, all that's like manually by spreadsheet and you know. Do you know, Fred? He's like 12 buildings away, and, uh, he only works two days a week. So, uh, good luck getting that info. Um, so, and it's outta date anyway, so like we, we've been pretty surprised at, uh, at, at how like big, uh, an opportunity there is to make people's lives better at all sorts of companies.

Mark Curphey: 15:18

I mean, two kind of related stories. So we're, we're, we're like tiny. There's ten of us, right? Um, but one of our frontend engineers has started using it. So, you know, when you put a code into CICD, Magic happens, right? So like, you know, and all of a sudden he's like, well I actually don't know what version of library is actually running in production. Like, I, I kind of know what was running locally, but I can't literally, I can't replicate production to go figure out what, what happens. So it's kind of interesting from that perspective and a lot of developers have that. When we were, John and I were doing the interviews, um, This was a great story. Like log, the whole Log4J thing when we first started doing them was, was kicking off and there was one person that was just like jaded by this thing because they literally had to send someone round trying to figure out what was running in production. So the SDA tool, of course, like runs on a repo and says, ah, you've got this, this version of this. And they have like, first of all, don't know which of those repos are deployed into, into prod. Secondly, the CICD is so complex that it wasn't just taking a piece of code, it was optimizing it, putting in containers from other things and then figuring out how to deploy it. And then even when they've got now hundreds of those things, which ones they go tackle first? Like they're dealing with this urgent, urgent problem. Um, and that was a really interesting one, cause literally they were deploying people to go figure it out. And they were like, you know, wouldn't you, you'd much rather be able to go query a set of metadata and figure that out in instantly. Like when we realized that again comes back to this like, what do I do now, next, or never? In order to answer those questions, you've gotta have loads of telemetry back. It's kind of like log files, right? Like I already wanna figure out what's going on. I've gotta have all the log files and they can go ask the questions, but the question may not be preconceived until I, until I know what I...

Robert Hurlbut: 17:07

So we have, um, an understanding about what Chalk is, a telemetry tool, and. And some of the problems it solves, but, um, is this also helpful for, uh, security professionals as well? And maybe what's different maybe for them versus developers and the data they're getting?

Mark Curphey: 17:26

Oh yeah, for sure. Like look, imagine you're scanning all your, all your repos with, with some SAST tool, right? Or you, you're running an SCA tool across all your repos. Like 90% of them you just kill in the first place. I don't even need to go look at them to figure out where they're important. Like I don't need to be scanning these things'cause they're not even important repos or not, not in prod. So like those are some of the important things. Other things are, look, I got an app in production. Like, I can tell from the, from the telemetry, this thing goes and goes and connects to relational database. Like, oh, that's probably managing PII, again, all that prioritization stuff. And then other types of scenarios, you've got a co, you've got an inventory of who owns the code. So now I've got an incident, right? That happens. Like I know that it's on this machine. I can then tie it back to this repo, which I can never do before, and now I also out the list of the owners. Great, let me go talk to them. But you can also do things like, for instance, a fix has already been put onto this branch. And it's a fix for this issue. Great. I actually know that someone is fixing this thing and I can track it and make sure that the fix is actually deployed to production and that it's gone. So again, that's sort of like closing the loop of the, of the, here's the incident. Now I go figure out how to make it, how to run it more efficiently or track down that the issue has actually been closed.

John Viega: 18:43

And, and putting on my, uh, cynical old man hat. Uh, like for people that really are just about ticking boxes, uh, that like the, the. Chalk is also really great for that too because like you can do first, like SAST scans, like it'll run Semgrep automatically on everything, collect SBOMs automatically for everything, digitally sign everything for you with, uh, with Sigstore and package that with the software artifacts wherever they go too. So like without anybody having to do anything if they don't want to, like, it can, you know, everything, including the key management can be like, fully like automated. Don't ever worry about it. So, uh, the, the, uh, you know, the like we're pretty agnostic about what metadata gets collected. It's make as much available as people want. And bundle it in a way where you're, it's, you can send it anywhere you want, but also you're always gonna be able to tie it back to the artifact. So the, like, the actual Chalk mark and the artifact will tie it back to the data that you collected. But you could also just like jam all the data you want into the Chalk mark if you really wanted as well.

Chris Romeo: 20:20

So

Mark Curphey: 20:20

you, I mean, you can do things, you can enrich the data. So like as an example, if I know this data is pushed onto an EC2 instance, I can then enrich it with cloud info. So I can say, ah, this code is actually deployed into Europe. As an example, or this code is deployed into this account, so that account happens to be, you know, shared services account. Again, once you've got that data, enriching it with kind of interesting for more context.

Chris Romeo: 20:46

Originally I was gonna, I was gonna describe this based on what I've learned so far as a, an observability tool for the CICD pipeline, but then based on John's description of, you know, generating signatures for packages and, and doing some of these other things, it's almost like a Swiss Army knife for the CICD pipeline. I'm kind of marketing on the fly for you here, but I'm trying

John Viega: 21:08

Yeah, well, it,

Chris Romeo: 21:09

to wrap my head around what is this thing.

John Viega: 21:10

And well, and it can live in production too. So for instance, uh, if you've got uh, like if you're doing a, like you build everything through Docker, let's say like Chalk can transparently wrap Docker. So you think you're calling Docker, you're really calling Chalk, and it, it will then, take your existing, if you wanted it will take your existing entry point, make sure that it still gets called and that its still paid one. But that it also spawns a version of Chalk that will report on the runtime environment. Uh, you know, even beacon periodically if you want to, and, uh, we're working towards, we've done a lot of work towards, uh, being able to send back, you know, uh, on demand data, like essentially like monitor for, for events in applications and push them as they happen. That's not in the first release, but it's definintely coming down

Mark Curphey: 22:26

Yeah, so, so scenarios there, like maybe, look, you're actually running Log4J and it's making egress calls out to something like, as an example, right? The telemetry thing Chris was kind of interesting. So like, so Joel kind of mentioned Semgrep and, and, and SBOMs and things like that is, you know, there's a whole set of telemetry you can just collect with a Chalk binary. But if there's other pieces that are missing, you can go spin up your own collection piece that you wanna do. I mean, you can define manifest metadata that you want to collect, that we can collect, but there may be other things that either you wanna collect in a particular format or sophisticated ways of doing it, like running a SAST tool or, or whatever. So, so orchestrating those and allowing those to collect that data is, is, is, is that piece, which is cool as well.

Chris Romeo: 23:10

So the million dollar question is, what is the performance impact if I run Chalk in production?

John Viega: 23:18

Uh, negligible, uh, in CICD it's also negligible unless you are like doing the SBOM or SAST scans, like those things can be expensive. But, uh, everything was written with, uh, performance and scalability of mind because like my last company, Capsule8, that was like our bread and butter was doing uh, detection and analytics in real time in production infrastructure in big tech companies. So like everything is, uh, uh, very geared at, uh, being as lightweight as possible. Now, like when we get to the ability to hook what's going on in an app and stream events back, you're gonna have costs that are associated with like, how much data you ask for potentially, but we know how to like, you know, make sure that, uh, you can cap those things well and handle it gracefully what you do essentially. So, uh, like all those things are, are, uh, but, but for what's there now, collecting data during CICD, collecting data in production, it is uh, ridiculously fast. We're using, you know, we're essentially like everything's statically compiled, uh, with a, you know, type safe systems language. Uh, and you're getting like C performance on, uh, on everything.

Chris Romeo: 25:00

What, uh, what, what platforms or frameworks are supported initially?

Robert Hurlbut: 25:04

Yeah, I was wondering as well,

John Viega: 25:06

Yeah. Um, so the intent, uh, like we're, we're more focused on Linux production systems at the moment. So like, be it X86 or ARM things do work on the Mac. They might work in like a WSL environment, but we are not really working on that. Uh, then it's a question of what kind of artifacts can we Chalk? Like, what do we connect to, to collect data from, um, the, the, like we, we've got basically any ELF executable, anything, uh, that is a, even a remotely common scripting language. Um, the docker world. There would some gaps in, uh, on the edges, but, uh, like, uh, 90% of what's common with like docker, uh, probably 99%, uh, we, we, uh, handle. And then, um, when you get beyond that, uh, there are things that. We're work naturally in a serverless environment, but we need to do more work that's like specific to the serverless environments. Um, and. Also like the, we have a way to do like the same stuff with JavaScript in the browser. That, and we think there's some good use cases for that, but it's not anything more than like a roadmap item at this point. Um, so, you know, it's still pretty comprehensive, uh, for like the Linux world. Uh, the data collection side, uh, there's a lot more we wanna do. Uh, but uh, like we're, you know, I think that a lot of the roadmap there is gonna be driven on what's most important by people and whether other people feel like they want to like their sleeves and help us with it. Because, you know, eventually we'll shift our focus to figuring out what we're gonna do as a commercial business instead of like just, you know, uh, building cool open source on investors' dime. So, uh, you know, the, uh, so at some point,

Mark Curphey: 27:51

from the podcast.

John Viega: 27:53

no. No, they'll, they'll, they'll live.

Chris Romeo: 27:59

So one more question about Chalk then. How does it, how do I get events from it? Like, is it, what, what protocols, what, what message formats is it sending outbound to get to advise me about what's happening in production? I.

John Viega: 28:12

So, uh, everything that it outputs is just like, uh, JSON, and then there are... gonna built in connectors for like an, oh, well, you wanna set up a place for, to h b post to or write stuff in an S3 bucket or log stuff to like a rotating log file. Like there're essentially, know, output sinks for all of those things that you can configure, sending, whatever, where you know, whatever data you want, wherever you want it. Always is gonna be in JSON uh, but, and you know, we'll probably end up adding more, uh, kind of sync types based on what people find valuable. Like, for instance, for object storage, we do S3, but we don't do any of the other cloud providers because, you know, we, we're not, we don't have credits on those cloud providers. So, uh, you know, but, uh, they should get done at some point. So.

Chris Romeo: 29:18

Okay.

Mark Curphey: 29:18

But you could ultimately dump it in Splunk or

John Viega: 29:21

Yeah, anywhere and and I'm,

Mark Curphey: 29:23

wherever you wanna go.

John Viega: 29:25

Yeah. We'll, we'll, I'm sure we'll end up building like more specific connectors, but the, like, it's just really easy to kind of generically get stuff out either in like JSON log files or via like just an HD post, you know, anywhere. So. Like things we would like to have specific integrations for, but we started with the, you know, the Swiss Army knife if that makes sense.

Chris Romeo: 29:58

Yeah. Cool. So let's, um, let's, let's change gears a little bit here and, uh, wanna get Mark's thoughts on this, the ZAPs move to the Software Security Foundation. Kind of get some perspective.'cause I think a lot of folks in our community are trying to figure this out, trying to understand what's happening right now between OWASP and Software Security Foundation and so, Mark, you're the source. So I wanted to get your take

Mark Curphey: 30:24

Yeah, no. Software Security Project. No, no, no. It's all under Linux Foundation and, and all that. So, you know, when I went around for the portfolio almost last year, it was kind of on this manifesto change. A lot of people wanna change, um, rightly so. Um, you know,OWASP does a lot of things really, really good and always has. Um, but it's very much kind of bottoms up and when that means anyone can go start anything. You know, 300 projects there, some of which are amazing, cyclone and ZAP and other things. Some of which less so, right? Some of which haven't been touched for 13 years as an example. So OWASP philosophy was like, that's it. Have at it. Let people sift through it all. I think that's challenging for a developer who we already know doesn't really care about security. Like how do they come to a place and find out these are the best tools for that. For the, for the job. That's not what, what OWASP does. Same with documentation and guidance, right? You got, you know, three or five of everything. So there was this kind of conflict right around what I thought and what a bunch of other people thought wanted to happen versus what the rest of the OWASP board thought or, or most of the OWASP board wanted to happen. The same was true with the governance model, right? So my take on it is that, and for the most part, all of the people in governance, uh, are basically vendors, right? And consultants. And so in general, OWASP people build stuff for themselves. And if you're a consultant or a thing, that's fine, you scratch your own itch. But when it comes to things like, could we build training? That is free training for the world. Not necessarily in everyone's best interests, right. To, to go, to go do that. Um, and so you've got a set of conflicts. And for my take, I would much rather have Google, Microsoft, GitHub, the big boys putting big money in to go move big rocks. And I, you know, some other people kind of said that that was a conflict. Well, I'd much rather have someone with lots of money having an opinion, then someone with no money having an opinion. That's my take. So, you know, when, when it's, it's very clear very early on that that thing was just not gonna change, right? And so, um, so basically kind of said, great, let's, let's go figure out, um, a different model. And there was this open letter from a lot of the, the, the flagship projects. And they basically said, look, you know, we, as we've grown, things like ZAP, as an example, we wanna go work on this thing fulltime, but we get no support from OWASP. There's no one managing our GitHub, there's no one managing sponsors. We're managing it all ourselves and we want to go build software. So, great. I, I can, let me figure out how to go create something which is different. And so absolutely your hope is, you know, OWASP stays the same and does what it is, what it is good at. And then the Software Security Project, a different governance model, a different model of how we're gonna do things. And the two absolutely, hopefully will, will work side by side and we'll, We'll, both, we'll both do great things. So there is absolutely zero, you know, intention, um, and in fact I worked hard to make sure that it doesn't collapse, like it can't, but, you know, there's definitely a bunch of problems that they have to go solve over there. Otherwise things are, you know, not gonna, not gonna resolve. And I think it's kind of, it's, it's appropriate to be open and honest about that as well. You know, the advantage of working under the Linux Foundation is that it, you have huge amounts of reach, right? So, you know, a CubeCon, you know, six, 6,000 people or whatever attend CubeCon. You've got OSCON, you've got OSSF, you've got all of these foundations with all of the big players in. And if you wanna expose software security to the masses, that's the place to go do it, right? The ecosystem already exists. So that's why it was a really great place to go, to go, you know, do it and make it happen. And we're still in the process of actually figuring out how that governance works, because we've gotta take the best of different models, right? We wanna make sure that the community can be involved. We wanna make sure that the big tech companies can be involved, but we gotta make, we gotta figure out like, what's the balance to make sure that you know, neither the, neither of the vendors, the tech vendors, the, the security vendors, security consultants can steer it in the way that they want, nor, and the big people steer it in the way they want. We've gotta make it conducive to people putting a lot of money in so we can do, do a lot of big projects. And we've gotta make sure that the things that we're gonna do are the things that are meaningful. So, you know, as an example, the top 10, OWASP Top 10 has always been criticized because it's driven off of scanning data, right? For the most part, not entirely, but for the most part. Well, that's circular causation, right? Like, You know, tool goes and finds these issues, then they become the most important issues. You then train the consultants to go find those same issues, and if you go talk to our CISOs, they'll say they're not their biggest issues at all. They are the most found scanning issues. So, you know, a good example of that is we're, we're picking off two projects initially. One is gonna be a top 10 driven by CISOs. It's like, what is the DevSecOps top 10? What are the top 10 things that we care about the most? So that, that allows consultants, vendors, and everyone else to align and say, let's go help you solve those problems. Versus the vendors saying these are the things you should care about, and the CISOs saying its not. So hopefully we can shift the conversation to, these are the most important things that people, that that, that the people that are trying to secure stuff are saying, Hey, let's go help'em work on those things. Um, and then the second one that we're working on is a DevSecOps Pipeline, a reference architecture. So, as you know, a lot of kind of people would say, ah, you like you need this, you need this. And it's very reminiscent of the early days of OWASP. These are things you should care about and et cetera. Well, let's get collectively people together and say, this is how you build a pipeline. These are the pieces that are in there. You've got version control, your cloud configs, you've gotta have secrets management. You've gotta have these different, different pieces in there. And then we're gonna build an open source reference implementation of that. So those are the tho That's kind of it. We are we gonna take our time and make sure that the governance model gets properly done, because if it doesn't, we'll, we'll hit a bunch of other things and make sure that all the people that wanna be involved can, from people that wanna put a lot of money in, people that wanna put their time in right the way. Yeah. Right the way up the spec. So, so yeah, that's where it is. And then with respect to ZAP, I mean, look, ZAP, they're amazing guys. Like that thing has so much reach that's like, I think they, they estimate it's like 70,000 people use the thing, like there were 10 million docker pulls last month. Like that scale is insane. And here was a tool that everyone is relying on, but was collapsing. And again, OWASP unfortunately, well, not unfortunately, OWASP, the OWASP leadership decided not to go figure out how to get this thing funded, either. Either decided not to or unable to go do it. So I did it. Um, and you know, we've got, we've managed to get onboard so far, Simon Bennetts, um, and Ricardo, who are working on it full-time. And as SSP goes, like, we're, we're gonna, we'll, we'll hopefully try and build a team of about 10 people working on it full time and, you know, fully open source, free for everyone. So everyone's gonna get commercial grade, open source tool. Now we've absolutely gotta find a way to make that sustainable because there's loads of people making money off of it. There are people embedding that into commercial tools and making millions off of it, and not contributing back. So, absolutely part of it as well is let's figure out a way to make sustainable open source projects financially sustainable. And then we can go apply the model to other things. Because we know there's a load of other people that wanna go work on them full time and then the tide rises for everyone, right? So yeah. So that's, I dunno if that makes sense.

Chris Romeo: 37:53

No, it's, it's actually very helpful. John, I think you're on mute here.

John Viega: 38:01

Yeah, I forgot I did that'cause my daughter was, uh, leaving for college in the background. But, uh, uh, one, one of the, uh, one of the things that I've been impressed with, with the ZAP team is, uh, you know, that they really are focused on doing the right thing. They don't want to go start the commercial company. They just wanna build their project. And, uh, if it's something like ZAP where the industry needs it, uh, you know, we, we need to have support infrastructure. Nonprofits like OWASP, SSP. Really, we need to make sure that people like that can be successful. That, uh, people that just wanna raise the bar for everybody else and help, uh, uh, if they're gonna do a good job, uh, can do that for a living and not, uh, have to feel like they have to go build like a billion dollar company. So,

Mark Curphey: 39:03

Yeah, I mean, I mean, to give you so many amazing things that the ZAP guys want to do. One example is, when you run a DAST tool, it's incredibly noisy, right? And you don't really know where to go point the thing. Well, imagine if you have a code base and you have an open source static analysis tool that knows where the endpoints are, and then you can go highly target the endpoints, amazing. Plus you go scan this from the outside and you know where the code back is'cause you know, you know all of those things like. These are things that they wanna do, but they were unable to do with the resources they had. And so, you know, everyone's gonna get all those sort of things in the right way, as John said. I mean, there are, there, you know, there are, there are commercially supported open source core things, Chalk, Semgrep other load loads of things like that, which are, which are great, but there are other things just frankly just need the money, and you know, there's a way to go get it for them. So, yeah. So we're able to,

Chris Romeo: 39:59

Yeah, it's always those folks are, I, I always think of them as so noble because like you said, like they could go start a company and they could, and, and people would line up to give them money. I. Because they've got, you've got a 10 year track record of this thing

Mark Curphey: 40:14

If

Chris Romeo: 40:14

for nothing with no money.

Mark Curphey: 40:16

a venture capitalist rocks up with 70,000 existing users and that like, exactly. It would just be like, you know, more, more startups would just beg for that

Chris Romeo: 40:26

Yeah. I mean, they just, they'd get a, they'd get a blank check handed to them. You just fill in, fill in what you need.'cause we know you're, you know, you're, you're, you know how to get things done for limited money here. Just fill in what you need.

John Viega: 40:36

And then when they, but if they go the ZAP route and choose not to do that, then they will end up feeling screwed because like successful projects like ZAP to Mark's point, there are like a dozen commercial products

Mark Curphey: 40:50

Mm-hmm.

John Viega: 40:51

use ZAP under the hood and aren't contributing back even to the code base. Nevermind, uh,

Mark Curphey: 40:59

Yep.

John Viega: 40:59

sharing some of the love with...

Mark Curphey: 41:04

mm-hmm

John Viega: 41:04

the people who drive revenue for them, so like it definitely

Mark Curphey: 41:08

I mean, I,

John Viega: 41:09

problem.

Mark Curphey: 41:11

I mean, look, I sat on a call. We, we've gotta figure out how to make it fair, right? If you're making money off it, you need to fairly contribute back into it. Uh, you know, I was sat on a call with, with a, I won't name'em and shame'em here, but I will name'em and shame'em if they don't, if they don't do the right thing. They are making$300 million a year selling their AppSec tools. And they tried to tell me on a phone call that they were being fair, contributing$25,000 back. Like, that's not fair. That's not okay. For you to go do that. Like, you know, and, and the, you know, the, the economics of open source are great. Like, you know, you build one thing, once everyone gets to embed it, it's a great community. You get to extend it, all of that. But if it's on the back of someone like Simon, Ricardo, Rick, who are doing all the work, and you're just, you know, draining them dry of all the money, that's not okay. We've gotta help figure out what that sustainable open source model is so that we can incubate and, and other, other things which are gonna, gonna be great for everyone. Yeah.

Chris Romeo: 42:08

Does SSP I mean, are you gonna, is part of SSP gonna be going after those folks if they don't do the right thing? Like, do you have

Mark Curphey: 42:17

Um, not,

Chris Romeo: 42:17

the ability to go after'em?

Mark Curphey: 42:20

I, I, I, I mean technically I'm sure. But I won't, and that's not the intent. But the intent is let's build a model. Where, you know, we can hold them accountable and we can, and we can make those people sustainably, we can make those projects sustainable and, you know, we can provide them with all the resources, you know, payroll, legal, IT management, so they can spend their time building, building software, which is what they wanna do. Um, you know, I, I don't know, there's, there's a different options we're kind of exploring, like, you know, one may be changing the license to GPL so these people can't embed it without doing it and dual license it, which allows every individual. I don't know whether that's just one option to explore, right. Um, you know, we'll, we'll, we'll have to look at all the different things, but we've, you know, we wanna, we wanna figure out how to, how to, obviously it's, you know, hundred percent gonna stay open source, there's no question around that not not being the case, but for the, we've gotta figure out how to make it financially sustainable and there's all these other people making money, so we've gotta figure out how to get the fair trade going and exactly how that happens is, TBD.

Chris Romeo: 43:21

Yeah, yeah. I mean, and you want to avoid lawyers and courts and things like that. Companies just need to do the right thing based on if you're building a business on top of an existing open source project. There, there's gotta be a, an equitable percentage that everybody can agree on that says if you're making 300 million from taking, commercializing an open source project as the core of your engine, I don't know what the percentage is, but it's, I mean,$25,000 into 300 million is a very tiny number. I don't know what it is off the top of my head.

Mark Curphey: 43:55

Right. It's rediculous

Chris Romeo: 43:56

I mean, it's not even covering one person. It's not even covering Simon working for SSP like that's, I mean, so there's gotta be, and hopefully vendors will come around to figuring out what, helping you figure out what that number is.'cause I think, you know, at the end of the day, I'm somebody that always believes as a vendor and I'm, I'm a vendor. I've had companies, I have new companies, like we should just do the right thing. That's what I try to do, and I think you guys are the same, you guys are the same mindset. Like just do the right thing. Like there is a right thing to do and there's a wrong thing to do. And the wrong thing to do is to give somebody$25,000 and then make 300 million on it. I don't think anybody, it's gonna be tough for somebody to make an argument with me that says, no, that's the right thing to do. And I know they, Mark, you said they tried to, but I'm not buying it.

Mark Curphey: 44:39

Oh

Chris Romeo: 44:39

Like

Mark Curphey: 44:39

two.

Chris Romeo: 44:39

I'm not buying it.

Mark Curphey: 44:41

Oh yeah. The irony is two PR people, two expensive PR people on the call. Right,

Chris Romeo: 44:46

There's, there's 25 grand right there, right in retainer and, and hourly fees. And they had to prep for the call and everything else, and I know I'm, I'm fun...

Mark Curphey: 44:54

but again, I, you know, my, my, my hope is, you know, ZAP, ZAP was an important project. We needed to do this for them now, but I hope we can use that as a blueprint for other ones as well, because it is really important. There are really, really important tools and really important technology that's being developed, and so we've gotta do it for the other stuff as well. So, so yeah, that'll be part of, part of what we're, what we're doing is, building the model, proving it outwards out. And then we can take it to other things and figure out how to get it done.

Chris Romeo: 45:19

Yeah my hope is OWASP wakes up, coming out, out of this experience, like for real. I mean, I'm, Robert and I are both lifetime members. Mark, you started the thing, John, you were there in the early days. Like we all love this thing. We don't, we don't have bad feelings towards it. And I, and I don't think, I mean, Mark, you don't either, like you, you,

Mark Curphey: 45:36

No no, absolutely not. Look, I had dinner with Andrew van der Stock last week, who I absolutely love. He just, he's just fantastic. But you know, the. I mean, look, the board needs, the board needs to wake up and, uh, they need to start listening and they need to start listening for, for a project and understanding what's going on. Like, and until that happens, unfortunately, nothing's, uh, nothing's gonna change. Yeah. I mean, SSP was, SSP was created because it had to be. I think that's kind of sad in many ways, right?

Chris Romeo: 46:02

And to everybody out there, there's four board seats open on the OWASP board in the upcoming election. So, you know, if you're listening to this conversation, you're like, yeah, things need to change. You've got an opportunity right now for a, a collection of people to go in and, you know, four, four like-minded people could shake this thing up quite, quite extensively. So, um, let's, let's, uh, transition towards our lightning round. Robert has become the keeper of the lightning round, and so Robert, let's take these guys through the lightning round please.

Robert Hurlbut: 46:32

Okay. Uh, so, uh, John and Mark, um, so I'll start with John. Uh, what's your most controversial opinion on application security, and why do you hold this view?

John Viega: 46:44

Uh, that it's nowhere near as important as the security industry thinks that it is.

Robert Hurlbut: 46:49

I.

John Viega: 46:53

So I, I, I don't know. I mean, just because I listen to people that aren't in security and, uh, what drives them is, uh, different and I think that, uh, application security is important, but has gotta be easy, otherwise it's never gonna be done well enough. Most people in the security industry are like as secure as possible at all costs, and that's not really the right answer.

Robert Hurlbut: 47:25

Okay. Uh, Mark, same question.

Mark Curphey: 47:27

Developers don't care about security. Like, and I, I think it's genuinely true. Um, I'd love them to, um, I've watched loads of people kind of say, Hey, we've gotta figure out how to make them. But I, I don't believe they do. I mean, it's one of the things, I have this fundamental belief, if you're gonna build tools, you've gotta build tools that provide developer value and security value. And by, by developer value, I don't mean like if they, they need to solve a side project of security, but developers won't, won't install security tools. Unless they're absolutely forced to. Because they don't wanna do it. So yeah, I think when we, when we, when we accept they don't care about security, you gotta figure out other ways to, to to, to get them to do it.

Robert Hurlbut: 48:08

Makes sense. Uh, mark, we'll start with you on the next one. Uh, what would it say if you could display a single message at a, on a billboard at the RSA or Black Hat conference?

Mark Curphey: 48:20

Uh, I'm gonna screw John. I'm gonna go"Chalk everything."

Robert Hurlbut: 48:25

Great. And John, same question.

John Viega: 48:32

Yeah, I, I don't, I don't have any message that I would ever putout there.

Robert Hurlbut: 48:38

Out.

John Viega: 48:38

It would probably just say yolo. I don't know.

Robert Hurlbut: 48:43

Excellent. Okay. Um, uh, back to you John. Uh, what's your top book recommendation for those interested in security and why do you find it valuable?

John Viega: 48:54

Oh, great question. Uh, nothing that I've written because it's twenty years too old. Boy, I like, I, I would say that, uh, Eugene Stafford's new book is, uh, really good and, uh, would be good for people getting in. I, like, I just have not spent a lot of time in books in the last few years, so, uh, that, that's probably the only one that I would, uh, would go to.

Robert Hurlbut: 49:29

Okay, great.

Mark Curphey: 49:30

My mine, mine would be two books that aren't security books, but I say this to security people and they're fairly old. One is Crossing the Chasm by Jeffrey Moore, which is a classic product management book. But what it talks about is how do you take things from early adopters and how do you figure out what requirements are needed so that you can get mass adoption on things? And there's so many lessons that can be learned around how that can be applied to security. So I think that's, that's extremely, extremely important. Um, the other one would be pragmatic marketing, probably for similar ways, but you know, John John's big pragmatic marketing person, but it really just teaches you to listen about what people want and how to go, you know, how to go solve problems for them. And I think a lot of the security industry tells people this is your problem, and here's how you need to go solve it Versus listening to someone about what the actual problem is, and how they would like to go solve it, and then building them a solution for that. And pragmatic marketing does a really good job of taking a step back. Yeah.

John Viega: 50:27

Yeah. I guess if the question's more, what books should security people read then The answer is definitely not security books. It's, uh, you know, how, how they can learn about the rest of the ecosystem they play in and, uh, like, you know, so certainly for product people. Uh, those are great books, but even for people that don't want to build products, I'm sure that, uh, you know, learning about, uh, the other, uh, like whatever the key techs are for other roles in the org that you deal with are really useful. Like, especially at the CISO level. I think that there's still a lot of CSOs that, uh, that are. Uh, not... That don't see themselves as business executives, and they would be better serving companies and customers if they thought of themselves, not just as like the security executive, but part of the broader leadership team trying to help the company meet its goals, not just meet the security goals. So.

Chris Romeo: 51:34

That's very helpful though. It's, uh, I, I love, I always love book recommendations that are outside of security. Because often security people, we spend too much time with the, the tunnel vision on what we think is important. But you know, there, there's so many lessons we can learn from other disciplines, which then you're like, wow, this really applies to security as well, because

Mark Curphey: 51:53

I got, I, I, I got one more for you then. I'm a, I'm a huge cyclist and, uh, it's about Team Sky and I forget what the name of it is, but you'll, you'll, I'll put, you can put it in the notes and the concept of the book is about how Team Sky, which was a cycling team, basically went, went to win the Tour de France in five years, and it was based on a concept called Marginal Gains. So when it first started, people would take it, would joke. They would take pillows to to races. So their cyclists had had better sleep and everything. And the theory of the book, which goes through is like, you make 1% change and you make it a hundred times and you're a hundred percent better. And I think that that is so true that people try and find all these big paradigm shifts and things, but if you're incrementally making things better, it doesn't take long before things are completely changed. That's a great book because it goes through the whole story of it.

Chris Romeo: 52:41

Okay, Okay, cool. Yeah, we'll find the, we'll, we'll put these in the show notes so folks can, uh, can find them. And, you know, I'm know I'm, you gave me a couple things to add to my reading list here

Robert Hurlbut: 52:49

Yeah, definitely.

Chris Romeo: 52:50

I appreciate that. This, this question is, is somewhat selfish. It's not really for the rest of, it's not really for the audience. It's really for us to read.

Mark Curphey: 52:57

We should ask you then. What's your book recommendation?

Chris Romeo: 53:00

Oh, man. I mean, I, what, what just happened here? Wait. This is our podcast. on, Mark. can't just, can't just

Robert Hurlbut: 53:06

Yeah, they can't turn it around.

Chris Romeo: 53:08

oh boy. Let's see. I mean, one of my, one of my classics that I'd recommend to people often is called Start with Why. Because it's, um, uh, that's his, the author's name is slipping my mind for some reason. But, um, it's, it's so often in all disciplines we jump into trying to solve the problem. Versus stopping back and saying, why do people care about this? Um, and so that, that's one of my primary ones. That's, that, that's, it's kind of a paradigm shifter.'cause it, it helps you to see there's different ways to solve problems versus just using what you, you already know by asking a lot of why questions. It's, it's a discipline to really understand, are we solving the right problem? Are, do we even understand the problem? And so that's, that's one of my, one of my all-time favorite ones. Robert, I'll put you on the spot now since Mark put me on the spot. Come on.

Robert Hurlbut: 54:00

Yeah, that's a tough, uh, I was gonna say, uh, the one that John mentioned I've been looking at, uh, recently, and so that's probably the one I would say, uh, at the moment that I would recommend, uh, people take a look at, uh,

Chris Romeo: 54:14

Yeah. Cool. Sorry. Uh, quick key takeaway here for our audience. 30 seconds maximum. Mark, you go ahead and go first. I. Key takeaway.

Mark Curphey: 54:25

Um, if you understand everything happening in your software engineering process, you can make informed decisions and decisions should be made on an informed basis. So, yeah.

Chris Romeo: 54:35

Excellent. John.

John Viega: 54:38

Well, that sums it up well, but, uh, for the conversation at the end, I'd also add, uh, you know, it's, it's really about, uh, getting in the heads of, uh, everybody else and not. Call it...[You] should not take an inside-out view of the world. Take an outside-in view of the world.

Chris Romeo: 54:57

There's your billboard too. You can put that on the billboard. Take an outside in, look at the world or inside out Outlook at the world. So, uh, mark, John, thank you for sharing your insights, your wisdom, your experience with our audience and us as well. We, we truly appreciate it. We look forward to, uh, a future conversation, uh, in some period of time for updates on where Chalk is or whatever else you guys are up to. So thanks for being a part of the podcast.

John Viega: 55:19

Thanks for having us.

Robert Hurlbut: 55:20

Yeah.

Mark Curphey: 55:21

thanks for having us.