Dan Küykendall -- Why All Application Security Products Suck

Dan Küykendall visits The Application Security Podcast to discuss his series "Why All AppSec Products Suck" and explain why software companies should understand the uses and limitations of any security tool. The series aims to highlight the limitations of each tool and to help users make informed decisions when selecting the right tools for their needs. In this field, there is no such thing as an expert; there is always something new to learn. Dan Küykendall is an industry leader in AppSec for 25 years with a total of 30 years in tech. He was the co-founder and CEO of NT OBJECTives until acquired by Rapid7 in 2015, where he served as Senior Director of Application Security Products until 2023.

Connect with Dan Küykendall:
→ Jeff Williams on LinkedIn
→ The Dan On Dev website

Mentioned in this episode:
→ The Dan On Dev website
→ Twitter (@dan_kuykendall)
→ Twitter (@Dan_On_Dev)
→ Instagram
→ Facebook
→ Fortify (OpenText)
→ Gary McGraw
→ Jeff Williams on LinkedIn

Follow the Application Security Podcast:
➜ Home
➜ X
➜ LinkedIn
➜ YouTube
➜ Instagram
➜ Facebook

Chapters:
00:00 Meet Dan Küykendall: Why All Application Security Products Suck
01:55 Yeah. we got a, a, a special guest who's been doing
14:35 A good, that's a good level of curiosity though, right
21:29 Okay. So I'm gonna, I'm gonna, I'm gonna come at this
23:38 That would be, we, we can call that your kind of
29:41 Let me, let me bounce my, the reasons I don't like
42:29 Hey, Dan, let's, uh, take a look at what we have
45:30 You remember, uh, TCP/IP illustrated
47:05 Yeah. DNS doesn't use Port 80

Transcript

Copy Transcript Copied

Chris Romeo 0:00

Dan Küykendall is an industry leader in AppSec for 25 years with a total of 30 years in tech. He was the co-founder and CEO of NT OBJECTives until acquired by Rapid7 in 2015, where he served as Senior Director of Application Security Products until 2023. Dan currently works on advising startup leaders and teaching software development leadership. He recently started incubation on a new startup. Dan joins us to unpack why all application security products suck. Spoiler alert, he doesn't really think they're all terrible, but you need to listen to hear what he really thinks. We hope you enjoy this conversation with Dan Küykendall. Hey, folks. Welcome to another episode of the Application Security podcast. This is Chris Romeo. I am CEO of Kerr Ventures and a stealth startup and also co-host of said podcast. Always super excited to be joined by my good friend, Robert. Hurlbut. Hey,

Robert Hurlbut 1:42

Hey, Chris. Yeah, Robert Hurlbut, principal application security architect and threat modeling lead at Aquia. And, uh, like you, excited to be here as well. And, uh, talking about our topics today.

Chris Romeo 1:54

Yeah. we got a, a, a special guest who's been doing a show talking about AppSec over the last number of months. Um, Dan Küykendall is, is joining us here, but Dan, we always like to jump in with no warmup for our guests. Literally throw you into the fire with your security origin story. How did you get started in this crazy world of application security?

Dan Küykendall 2:16

Yeah. Well, uh, good to be joining you, Chris and, and Robert. So thanks for, thanks for having me on here. Uh, origin story. Geez, I was. Fascinated with tech through like high school. So this was like in the late eighties, right? Uh, early nineties I was kind of, you know, junior high and high school. We had some computer classes. It was always fascinated. Uh, but I was a poor kid growing up, so didn't have access, uh, really to much in the way of computer technology until later. I had a, a roommate, uh, in one of my first apartments that had a computer and let me kind of just tear it apart on a regular basis and rebuild it and, Uh, and so I learned a bunch. There was a, this is back in the day of BBSs, right? So before the internet. And, um, and so I was getting on this local BBS and the, the admin of it, you know, I offered to help. Like I'll just vacuum right? Like, whatever you I can do, and just to learn, um, show me, right? And so he, he was helping me, so I kind of got on as like a moderator to the BBS. And then there was an online gaming system, uh, or gaming environment called The Sierra Network or The ImagiNation Network. And uh, and so there was a game on there that, uh, this game Yserbius, that I loved playing. But you would, this was back when you'd pay, I think, was it by the hour or something? By the minute. I don't forget. I think it was by the hour. And, uh, that it'd be a very expensive endeavor to grow your character. And so there was another player on the BBS with me that was actually like an actual developer. And so we were talking about like, oh, this sucks, you know, having to spend so much money to play to build up your character. And he had kind of been looking at it and he had me run some tools 'cause he was, he was running tools on his machine to see what was what, like how the system worked. And we've basically found that our character stats were saved on a file on the computer. That could just be manipulated. And so he created something called vitamin F and, and it could boost your character. And that just blew my mind, right? Like, oh, it's on here. Like, it just trusts me to like not manipulate that file or not have the skill to. So that kind of really opened my eyes to, to the security space and uh, and that's really everything else was kind of born outta that. I ended up at a job at UCLA got as like an IT, like a, um, I was a set, I was AV guy, I was like setting up microphones and and, and projectors for the, this, uh, these professors that were going out and, and teaching schools. Um, and so I ended up at the, went next to the Unix admin at the university. They stuck me in the library and information services building. High school dropout. Had no reason to be there. But this guy, the IT admin of the university was really kind and would share stuff with me constantly. So I was learning as the internet was coming out. So everything was just demystified. Right. And, um, and I still didn't understand it until there was a, a tool, a program. Uh, we were trying to take the BBS to the internet and we were bridging it. It was like a Wildcat server. And so we had done a bunch of stuff. I was using Linux to like do the DNS and I was starting to use Sendmail, and then I stumbled on qmail. Was it qmail? I think that's what it was. Yeah. qmail. And, uh, the way that Dan, I think it was Bernstein, I think that was his name, that was creator of it. The way he was building it and the way he created each separate piece to not trust the other, like, why, like, how's this happened? So I actually, like, I started convincing. I wasn't really much of a developer yet, um, but I started convincing him to that I would like, how can I help you? I basically found developers hated writing docs. So I actually got my start in Lin in the Linux community writing help docs. And in return for them answering my questions, I would write their their man pages right, and detail everything out. And, um, and so when I started seeing how Dan was doing things, I was like, well, that's fascinating. Like this is, like, why is he doing it? Like blew my mind. And then the next step of like, oh yeah, the other systems aren't doing it and here's the problems, right? And so I started digging in and um, so I did some open source work for a while. Um, Was cool. I was a speaker at the first open source, um, conference in San Diego Right. The, uh, that the O'Reilly Yeah, the O'Reilly publisher put on. And uh, that was cool. Got to meet a bunch of people and, and that demystified things as well. 'cause it was like everybody was a bunch of smucks trying to figure this out. And I think that's one of the things that, uh, a lot of people go into this space and they're intimidated by like, oh, there must be like super smart people building all these things. Like, no. Just smucks, just like the rest of us just happened to be writing that piece of code. And, and so as I started look learning, reading code, uh, it, I, I just kind of fell in love with the, the problem space and, uh, and so like Kevin Mitnick's story, um, during that, like him being arrested and all that, 'cause I was like hacking my neighbors at that point is like when cable modems. Came out and you were still on the same land with everybody right. Before they really natted everybody. Right. Um, and, and so watching the, the saga with Kevin was interesting. So then it kind of put me on like, I better find somebody that will actually like, let me like be, pay me to do this so I can do it under the veil of legitimacy. And, um, and so started kind of following along and, and got attached to some interesting teams, ended up at Foundstone. Uh, and worked on the, on the foundscan product. And uh, and that was a crazy place to be back then because we had some of the best researchers out there, right? It was really, I mean, if you go back into that space, there's generally, like AtStake or Foundstone, right? You come from one of those two worlds, right? And, and Foundstone only bundle EI and all those kind of, you know, whatever. But, uh, so I came from the Foundstone legacy and I just actually ended up there. Not even knowing what they were. It was, I was looking for, I wanted to get into development. So I applied for a php developer job and they're like, do you know who we are? I'm like, no idea. And, uh, ended up being foundstone. I was like, just, you know, felt like God kept leading me into every next step. It was just amazing. So that's how I ended up in it. Ended up separating from Foundstone to build NTO and uh, and I was really fascinated by the web security side of things. So that's where I've really spent my time since 2002, is purely on AppSec.

Chris Romeo 8:53

Yeah, that's such a good story and uh, You know, you're from the same vintage as, as Robert and I, and so of course you've aged a lot better. Just for those watching on YouTube, you, you've aged far better than either Robert or I have. Um, But yeah, I mean, just, you're making me, this is a, I've had a lot of guests recently that have just been making me nostalgic and you said Wildcat BBS. and trying to bring it onto the internet. Like, I mean, I was a BSS guy back in those, that same timeframe. Um, and so I got a, you know, I learned a lot from BSS and running A BBS and, and that type of stuff. Just, it's a, it's a great foundational layer as a starting point to, uh, to tech that just, it's, it's gone by the wayside. Like, it doesn't, I don't think any exists anymore, but I need to, I need to look it up and see if there's like still one I could dial into just to see, see if I could hear that sound when that modem would be, you know, you get that sound and we can all hear it in our heads that lived it. Anybody that's born after, you know, The year 2000 is like, I don't know what you're talking about. The internet does not make sounds when you connect to it, but we, we lived it. We did it. So, um, you mentioned Kevin

Dan Küykendall 9:57

connecting to it is almost absurd. just, it's always there.

Chris Romeo 10:01

Yeah. And then, Yeah. it's just, yeah, it's, it was such a different world. We, we grew up in, uh, as to as technology people. Um, you mentioned Kevin Mitnick, and I know you did a video recently about how you were able to teach AppSec to Kevin Mitnick and, you know, with Kevin Mitnick passing, uh, a number of weeks ago. I thought I'd, I'd just love to hear that story firsthand right here from you.

Dan Küykendall 10:22

Yeah. Well, as I mentioned, I had been following his journey in 2600. Right. And, um, and so super aware of him and. Uh, it felt bad for the way he was being persecuted. Right. And like, I mean, he did things wrong and, and whatever, but, uh, and I don't think he, he ever like said he wasn't doing something he shouldn't be. But the, the punishment, I don't think fit the crime. And, uh, and so anyway, I wasn't really, I kind of lost track of it. I was, we're running NTO at this point. We kind of had launched it and, uh, had launched our NTO Spider product at that point. And, uh, our sales guy, like I was, I was kind of a key man, so I was kind of doing a little bit of everything, right? If you're a startup founder, you kind of know what that's like, um, and you're, you're just doing everything. I was looking at the sales emails and one of them came in and it said it was from Kevin Mitnick, right? Kevin at Mitnick Security or whatever it was. And I was like, uh, okay, what's this? So I'm like, you know, Hey Carl, I'll, I'll, I'll answer that email. And uh, and so I emailed them back and, Sure enough, it was the actual Kevin Mitnick. And so I was like, you know, I guess I was probably awestruck at that point, right? And like, wow, this is like, this is crazy. And so he wanted a copy of the product and I asked, I didn't, I hadn't, I kind of knew he was outta jail at this point, and I think he was under supervision or he is just kind of finally getting out of his, uh, supervised release. And so, He was telling me that he's trying to get up to speed on, on web security or you know, at the, yeah, at the time it was web security, right? He was trying to get on up to speed on web security. You know, that stuff really didn't exist when he went in. Right. That wasn't really a field yet. And so he had really missed out on a lot and I just thought that was fascinating. So I had, I offered, I'm like, Hey, if you ever want some, Help. You know, if you ever want me to show you anything, I'd be happy to. And I, you know, you say that to a lot of people and usually they don't take you up on it, especially didn't think he would, but he's like, absolutely. If you, if you want to help me, I would love to. So we ended up setting up sessions to like sit down in the evenings and just like go over stuff. So couple days a week, uh, for a couple months, we would sit down and I would teach 'em another AppSec thing, right? Another web security thing. Start off with, SQL and then cross-site scripting and really the fundamentals of why those, like, how those happen, right? Not just the payloads, but the uh, the understanding. Um, and he was really like quick learner and, um, very curious and very humble. Like you just, it was, it was a neat experience I didn't expect. Right. And, uh, and we just became friends. Just we'd chit chat, we'd keep in touch. Um, Me and my wife would go meet him in LA for dinner whenever he was around, or, you know, not all the time, but when, you know, he was, he would come in and out and we would go hang out. Uh, you know, he had health issues throughout the years and, um, you know, he ended up moving out to Vegas or staying in Vegas 'cause of his mom and taking care of her. And he was just like, you know, there's the persona, but just like when you're with him and, and I think I started, you know, watching LinkedIn and watching some stories after he passed. Uh, I. You would see that, but like when he was here, you know, it just like was this persona and I, you know, some people would like, make fun of him, and some people were like, you know, uh, inspired by him. But, you know, he, to me, he was such a cool guy and just really like, chill, humble, um, and very curious. Like, he always was asking the next question, you know, um, there's that, uh, like the five, the five why's I think for what is, what they call it in the, uh, Uh, lean the lean startup or whatever, right. Is like five. Why's that? That was Kevin. He was always like, okay, why answer that? Well, why does that happen? Like, 'cause the, you know, he was like, he was always like, you know, I don't know, like the five-year-old that you're constantly answering the next question.

Chris Romeo 14:34

a good, that's a good level of curiosity though, right? Like, and it's a good reminder that everybody in our industry, there's people behind here. 'cause there's far too often in my career of watching what was information security. And then as I moved into AppSec, it's like, guess anytime you have a group, groups of people, you have some amount of drama, but I've never been one, like I'm life's too short for drama, but like it's a good reminder like you're telling these stories that are, that are, that are showing that Kevin Mitnick was just a, a regular guy like. If people, you know, who had feelings and, and cared for people and did these things, it just, it's a good reminder for all of us that like, we're we're just people. There's, we're just a collection of people. There's really nothing special about any of us that are in the upset community. Like, oh, yes, you might get to speak at a conference. Woo hoo. That doesn't make you any better or anything than anybody else. Like, you know, I've been around for, this is, I think I'm 26 years in cybersecurity and I am just, I am never, I'm not even amazed by the things that I can learn every day. And then that's why Robert and I started doing this podcast 'cause we wanted to ask people questions and learn and, and, and continue to grow in our knowledge of this because there is no such thing as an expert. I don't care what anybody says, like there are no experts. If you really want to go to court and prove you're an expert, you'll get beat because somebody will, will, will make a case about what will, what is this? Well then how does that work? Oh, you don't know that. You're not an expert. Sorry. So, um, that was my soapbox moment about, uh, people in security. So, Robert, what did, what did you want to ask next?

Robert Hurlbut 16:06

Yeah. So, um, you, me, you mentioned Foundstone. I remember that. That's a name I haven't heard in a long time and their products. Uh, but you have a series, uh, that you put together Why All AppSec Products Suck and, uh, can. you explain the motive, motivation behind that series as well as, uh, what prompted you to create it and the content.

Dan Küykendall 16:28

Yeah, and I apologize. We got lawnmower, of course. Um, so the, uh, so, so now I'm, I'm no longer at Rapid7, and, but while I was, I had actually put together this talk. I, I've, you know, I've done conference talks throughout the years, right? I'm always, and you know, when you're doing a conference talk, anybody that's done this, you know, you're trying to like, come up with your, like, your hook, right? And, and you're trying to make something interesting. And so, uh, I had been putting together kind of a comparison of the different tools. And as it is my normal want, I'm gonna try to figure out an angle to make it kind of fun, right? And so I kind of came up with this idea of why all AppSec products suck and, and then I. So I kind of restructured my, my conversation around that. And it really, my goal was like, as when I, even though I'm selling it, I was selling a DAST, right? That has been our primary product. Um, and then we acquired a, a RASP technology. And so we were kind of selling that too. But, uh, I just, my natural inclination when I'm dealing with customers is that I, I, I hate being like, salesy about my product. I would rather try to be like a trusted advisor. Like let, like tell me what's going on in your org, what are your problems? And then, and, and let me see if I can help solve it. And some of the solution may be our product and sometimes it wasn't. Right. And I would have conversations where I'm like, this is not what you need right now. You need a SAST, for instance, or whatever. Right. Um, and. So over the years, I'm always looking at all the new technologies, all the new trends, and, you know, keeping up with what's going on. And, uh, I, and I felt like people don't always have, people have a conversation from a vendor that tells why their product category is so awesome and better than the other ones. Right. And so I just want, I just, my inclination is to be a counter to that right? And so I, I just, yeah, that's just my natural inclination, so I. Came up with the idea of like going through every product category and writing out the strengths and weaknesses. And it's not about actually, and, and if you watch the videos, it's not necessarily and or the conference talk, it's not about entirely beating up on it. But if we don't, if you don't understand the limitations of a given tool, if you don't understand its strengths and weaknesses and really understand its both of those, you're not gonna pick the right tools. And uh, and also the idea that one tool is gonna like solve the, you know, be your solution. That's nonsense. You, you need a collection of tools that work on the different things that they actually can help with. And so that's, that's been the overall premise of, of the series. And it was a lot of fun. Uh, I took the conference talk itself and if you kinda watch the videos, you see the evolution of that. I, I, what happened is I, I did the conference talk and the very first time I did it, uh, it was kind of fun because, uh, Jeff Williams was in the room. And, um, and so like I, you know, I've known Jeff been around, you know, been in the industry for a long time, right. so we all know each other. And, uh, and so I, it was funny, my wife actually happened to be sitting in on the conference. Uh, she'd come with me 'cause it was in Santa Monica on the beach. And so she was hanging out with me. And, uh, and so I started kinda like, and I knew that when I saw Jeff, like, I'm gonna have to engage with him throughout this, right? And so I was kind of calling him out and we had a good con, you know, he was like, Bantering back and forth. And, and I looked at like our family chat later, and my wife's like, oh, your dad has a heckler, right? Because she didn't understand gen, no context. Right. And uh, and so like as I got to the IASt part and the strengths and weaknesses of the IAST, you know, he's like, well, that's not true or that's not fair So, um, it was good. It was fun and I enjoyed that. I was a blast. Um, so when I left Rapid7, I was like, okay, I wanted to start some YouTube. I wanted to start some, uh, Doing my, my own content and my focus, really, my goal is to be focused on dev leadership. That's something I'm very passionate about on like leading engineering, leading software engineering teams in particular. There's lots of leadership stuff. There's lots of, uh, engineering, uh, project kind of management, but like leadership, I don't, I don't see a ton of that content out there. That's really the goal of my channel. That's why it's called Dan on Dev and not like AppSec Dan or anything like that. Uh, but I had this content and I'm like, Hey, what if I cut this up into pieces so I could practice getting stuff on YouTube, right? And just learn, practice the editing process and, and uh, and it kind of like grew way beyond what I thought in the conversation was awesome. You know, the engagement on LinkedIn in particular was a lot of fun. Uh, but uh, yeah, it was just a breakdown. It was just taking my conference, talk into separate pieces and, and zeroing in on each product category, episode by episode.

Chris Romeo 21:29

Okay. So I'm gonna, I'm gonna, I'm gonna come at this from a different angle now, now that I've heard you kind of describe what your, what your series is. Okay. So we're gonna parachute Dan into a startup company. Okay. And so you don't have unlimited resources. You're not one of those startup companies. You, you didn't get a $500 million series A check attached to you, so you have some constraints on money. You can't just, you can't just write blank checks everywhere, but you have money to spend. What are your top priorities from an AppSec tooling perspective in prioritized order? So what, what would you do first? And give us a couple of those categories and then give us some rationale for why, what the benefit would be for, for that company, for that small startup.

Dan Küykendall 22:17

Yeah, that's a good question. Uh, I would probably focus on, uh, figuring out the most efficient way to spend on training content for your engineers. Because if you can, if you can solve some of the, uh, the issues, like, especially if you're that early, That's where you have to solve things. That's where you, you design in security. And so if you're really spending time designing security into the product, a lot of the other stuff won't happen, right? it just, it won't be a thing. Um, so I would, it would be a lot more about taking time than actually spending, I think taking the time to sit down and talk through design, talk through the security implications, make sure that... you may have some, like hotshot young developer, does he know anything about security and the implications of his design choices or is he just picking up a toolkit and, you know, uh, knocking out a quick UI and all that sort of like, it would be a lot. I mean, you could, you could do it for free on YouTube really. You could just, you know, I don't know that I would be spending that much in dollars to a product company. I would be spending dollars on, um, salary time to spend time learning and go through the design process that way. I don't know if that's a great answer, but that's kind of, that's how I would approach it.

Chris Romeo 23:38

So that would be, we, we can call that your kind of, your first priority. But what about, what about, um, if the DevOps team is saying, Hey Dan, we, we, we, we think after we took your training or your, your focus on training, we think we should, we should add some of these AppSec tools into our pipelines, but we're not sure which ones we should start with. So if you had to give us like two and like, like priority two and priority three that came from the tooling suites, what would be the first two categories of tools that you would add in for this company?

Dan Küykendall 24:07

Yeah, I guess, I mean, the way people build today, I would probably grab an SCA product so that I could make sure that the packages that I'm, that we're pulling in, That we, we have some understanding of what's coming into our project. So I would probably look at an SCA tool off the bat and um, generally, again, this kind of depends on what language you're using and all that stuff, but I would probably, the next thing would probably be a SAST is I would probably, especially if I'm that early, I would want to, to be making sure we're monitoring the code as closely as possible. And, and that doesn't necessarily mean like a Fortify. I would probably avoid a, a big huge package, like a Fortify. I would try to look at some of the smaller vendors that are specializing in the language that I'm working in, uh, because I don't have the large company problems at that point. Right. When you're a large company, you have All these different program teams writing in code in different languages. Your choices are gonna be different than if you're a small startup with like one code base. And so I would be operating on like, this is one code base. Let's focus on this for now and, and try to keep my spend down.

Robert Hurlbut 25:23

All right. Switching a little bit on some other tools, um, what are your thoughts about DAST, and this is sort of a loaded question for Chris. But, uh, curious to see your take on DAST.

Chris Romeo 25:36

I, I didn't know that you started a DAST before I added that question into the mix. So,

Dan Küykendall 25:42

Oh, I thought that was the intention. I thought you wanted

Chris Romeo 25:44

wait. I can't

Dan Küykendall 25:44

...wanted to beat my own kid up. Really Um, okay, so, well, one, I've spent nearly 20 years building DAST Right. And building DAST products. Um, so I do think they, I really do like DAST. Within what they're able to do. Right? The over expectation tends to be the biggest problem. Uh, but I think the, the nice thing about DAST is that you are able to get a comprehensive like understanding of the application that is actually running. Sometimes when you're looking at code, you're looking at a bunch of stuff that isn't even necessarily executed, right? There's all these different challenges. With a DAST, you're, you're dealing with the running applications and, and you could test things, uh, for real. Uh, so I, the DAST and I can, you know, this could be its whole, like we could have a whole episode on just the DAST and I had a, you know, a video on it too. But it, it's able to look at things, uh, in some ways a lot better because you're dealing with a real application. The challenge that it has is that it doesn't understand context. And so, um, you know, it could be clicking on different pages, but if it's like looping through an inventory list, it could go on infinitely, right? Because it doesn't understand, the context doesn't understand that I'm just, you know, yes, the URL changes a d ...different page but is it really a new content? And there's things that we've solved, we, we've done things to solve that. Um, looking at like what sections of the DOM actually are changing from page to page, and we create these signatures and we, we have all these techniques for trying to avoid those infinite loops, but it is like, these are these rabbit holes that, that DAST is gonna go down. Um, and then the other thing is when it's trying to, um, populate like forms with valid looking data, that can be very difficult 'cause it doesn't know what, what's actually being asked or, so there's all these like, and then there's, there's workflows, like even going through a shopping cart. If I see these pages as I'm going through the crawling process, but then I like attack 'em out of order, right? If I come back later and just like jump right to attacking the billing page and I didn't like add anything to my cart and fill in the shipping page, I can't attack the billing page. It's gonna reject me out of hand, right? Um, and so there's all these things that you have to deal with and uh, and that workflow, let's say is pretty common. So I could write detection and, and support for that workflow, but. The next workflow that's completely custom to the product, how is the desk gonna understand that? So it needs training for those things. Um, I tried to approach the, the effort of like on the assumption that the user is not gonna give me anything more than a URL, maybe credentials. Um, and that's gonna be like 98% of the time. That's all I'm gonna get and I have to do a good job. We would do a lot. We try to solve a lot, but there's limitations. And then I would say, okay, for those other 2% that are actually gonna gimme some training data, I would try to make it as easy as possible. Give me any format you have, you have a Burp suite log. Cool. You have, you know, PERS proxy back in the day. If you have, you know, zap proxy, whatever you have, if you've got selenium scripts for your unit tests, anything you can. Because I, the, the, from a DAST perspective perspective, it is desperate to understand a little bit more about the application and the actions that users take to be able to do effective testing. So I like 'em, but I'm very aware, of their limitations. I don't know that there's anybody, I'm more aware of the limitations of tasks than I, than than myself and, and some of the people on my team. Right.

Chris Romeo 29:40

Let me, let me bounce my, the reasons I don't like DAST anymore, and I'd love to get your take as somebody who's thought about this a lot, a lot more in depth and built a product around the whole solution. So the, one of the biggest challenges I have with DAST is I don't think, I mean, DAST is not compatible with DevOps. You can't go, you can't create a dev a a fast enough. You can't do DAST fast enough in a DevOps world, at least not in the pipeline. Is that okay? Because like some people try to, and it's like, I mean, zap did this like one minute scan option where like it would try to do as much as it could in one minute. I'm like, what does that mean for anything? Like that's, that's just letting somebody check a box. If you're doing a one minute scan and you're not doing a complete

Dan Küykendall 30:23

operation. No, these scans could take a long time. Like I, we'd have 20 hour plus scans on a regular basis. Yeah.

Chris Romeo 30:30

And so that's, and that's one of the, and that's another challenge that I see in, in DAST, is that lifetime scanning time. Um, and I just personally, I never had any success with DAST, with modern web applications. Like with React and Angular front ends, which is anybody that's in the startup world, if, I mean, you're not, you're just not building anything that's not using react or Angular as your front end. And so I just had very little success getting any measurable, many actionable data to come out of a DAST. And Robert knows this story, but I, I was forced to use a DAST as a procurement requirement and I fought tooth and nail in the procurement process. I'm like, no, you don't understand. I understand pipelines and security, and we're doing SCA, we're doing, uh, SAST. We're doing secure code review, we're doing threat modeling. We're doing all of these things. I don't need to run a DAST. Oh no, you have to have a DAST if, if, if we buy your product you have to run a DAST. So that's what I, I had to go out and I looked across a number of different options in the industry and tested them against our React based front end and Ruby on Rails backend, and none of them provided anything but garbage. I. So I'm like, I gotta spend money now and set this thing up to run once a week to check a box. 'cause I believe if I agree from a legal perspective, like I agreed to it, I signed the paper, my name was on the paper. I'm like, I'm gonna do it. But I'm like, it was frustrating that I wasn't getting any value outta my money. And you know, I mean, you've, you've Dan, you've been in the startup game as well, like cash is king and, and it's, you know, you don't want to throw, you don't throw money away in a, we were a bootstrap startup, so it's even like, like every dollar was very important to me. Uh, because I didn't have a million dollars in the bank to do it. And so that's, that's where, you know, some of the, the origin story of where my dislike of DAST is coming from. But I mean, how do you, how do you, what, what, what are your thoughts on that? Like, am I, am I, is there any l any reality to what I'm saying here, or, or what do you think

Dan Küykendall 32:21

Oh, totally valid. Totally valid. Um, so it depends on the DAST product you're using as well. So DAST has had a difficult time evolving. Okay. And we try to stay on the, on, on that effort and like I. Man, most of the DAST that don't have their founders still, and the, and people that are really passionate about evolving the tech stack, uh, they fall behind. So, uh, yeah, I had this conversation with Caleb when he left, uh, spy Dynamics or HP at that point, right? He was like, when they, when when you leave, you know they're gonna, they're gonna, I'm like, thanks for leaving. 'cause now that product's gone And he's like, nah, I left him a good roadmap. I'm like, if you're not there to help, make sure that they execute. It's not gonna happen. Sure enough. It, it's, so in the original, like in the earlier days, let's say the early, like two thousands to the mid-teens. Okay. Um, it was a lot easier to do a DAST because the pages were a lot more static. The client side was not very heavy. Right. And so you could crawl very well. You could actually crawl those pages very well and then do the tests. Okay. Um, I remember it was, I think it was 20, 2012 maybe? Yeah, 2012. Uh, 2012. There was, uh, I got a phone call on a Friday afternoon and uh, customer was saying like, I tried to run the scan, nothing's happening and what's going on? And so I'm like, okay, well show me this. Show me your scan. Right. And. We're like on GoToMeeting. So I had him spin it up and he ran a scan. It starts and stops like 30 seconds later. Right? So I look at the page myself, I'm like, oh yeah, it's just an HTML page that's loading a Flash app. And that Flash app is doing all the rest, right? And it's communicating with AMF to some backend. And he is like, he is like, okay, so what do, I'm like, yeah, you're gonna need like a pen test because a DAST scanner can't do that, right? Like, you can't deal with that client side and. Um, and we also didn't know how to deal with AMF at that point, right? Like this is a whole different data format. We're used to the name equal value pairs of a get and post. Like that's how the DAST scanners were all built, right? That's all how we were all designed from the, from the ground up. And so, uh, he's like, well, what do we do? And I'm like, we gotta get a pen tester. He's like, we don't have time. This is the feature of their Super Bowl campaign. This was like PepsiCo and they had like some stupid like social media thing for this, uh, this Super Bowl campaign. And so I'm like, okay, well lemme take a look at it. I grabbed Burp, grabbed an ambient f decoder, started like parsing the things down, injecting SQL injection into the payload, you know, into it. Repackaging it up, sending it off. Sure enough, wide open. I'm getting PHP errors. I'm a good PHP developer mentioned that earlier. Um, and so I started looking at, I grabbed the AMF PHP library, see how it works. Like, okay, he is just trusting the objects. He just, you know, whatever. Get your developer on the phone. We ended up going through and fixing it. I showed him like, I'm like, Hey, look at this line of code. You're trusting the data from this object. And he is like, oh, you know what's happening? And um, so I show him the errors I'm seeing and, and uh, so we went through and fixed it and whatever, but I, I had to go back to the team and say, Hey, like after, watch the Super Bowl, gotta a kick out of it. But it got me thinking like, DAST is gonna die if we don't redesign all of this, we are gonna be obsolete because we couldn't handle the data formats and we couldn't have handle the heavy client. So we actually took a step back, basically redesigned the entire scanner. We separated the data format from the attack payloads 'cause we were starting to like wedge 'em in. We're like, oh, if we wanted to support like JSON every attack module has to get updated and like, this is a nightmare. Let's redesign it, let's separate it. And we call this like a universal translator. So we had to, everything would get brought into a standardized object, but that thing knew how to deconstruct and reconstruct every data format. And so we redesigned around that. We actually got ahead of the game. We start, we could handle, we actually could handle AMF. We could handle, uh, JSON XML. Uh, we had RFPC version of XML. We had, uh, Google Web Toolkit when it came out and it had its weird pipe limited format. Um, we would add in support for all these different data formats. So that solved a big piece of it. That got us a long way, like when we got acquired to rapid into Rapid7 and we were kind of doing things, but then client side: React. All of a sudden now the client is so heavy and like React in particular threw us off at first because we had a model where we would actually grab the DOM or grab, you know, like load up the webpage, put it in a real browser object, and then go through all of the elements that have events on 'em and trigger the events. See what would happen. We did that on, on a React page, and there was like, nothing had events. There was just one event on the doc, on the, on the document. And they had this shadow DOM, so then I had to, we had to go figure out like, what the hell's this do? And so like, if we could go learn the shadow DOM and we learned how to like translate it back to the real DOM elements and then trigger 'em again. And so we could start getting somewhere, but we started having to do that for every framework and then, you know, React changed its shadow DOM and then we had to go and try to catch up with that. But we were so like constantly behind because it's, it's a difficult problem when you like Until there's enough adoption for it to become a problem that's worth investing development time into resolving,

Chris Romeo 37:59

Mm-hmm.

Dan Küykendall 37:59

uh, we wouldn't do it, right? And so you're constantly like trying to figure that out, but by the time then you go, oh, well now it's gotten so big, everybody's using it, and now we've gotta try to catch up. And so, uh, I don't know the timeline of when you needed that, but you probably were dealing with the DAST that hadn't figured out how to deal with React yet, and, and it was basically pointless for you, right? It's like, Almost doing nothing. Uh, so, so totally valid. And that gives shed some light on the, the actual design challenges. Um, but also your use case is wrong for DAST. Okay. In your model, I'm not sure I would be prioritizing a DAST. Um, now maybe doing a DAST, you know, maybe doing a one-off here and there just to like give it a final sanity check would've made sense. So even if you would've, like, you know, a, a once a year you do a $2,000 scan from some, you know, from Rapid7 or, or um, uh, nets Sparker, right? Or invicti, right? Do a SAS offering version of that. Run it once a year or something would probably make sense just to give you some extra sanity check, but it's not gonna work in your pipeline very well. Uh, especially if you're doing React or you're doing a lot of like really modern. UI design, you're gonna constantly outpace the scanners. Um,

Chris Romeo 39:19

So is this, is this why there, there's, we've seen the modern kind of API scanners have become kind of a its own class of technology, and sometimes people put those with DAST together, but are they doing, are they fo, are companies focusing on just scanning API because of the complexity of, of keeping Yes, they're

Dan Küykendall 39:40

trying to avoid the, they're trying to avoid the client side. Complexity. And, and, and they're also so like we, we changed to handle all these different data formats. Okay, so we can handle the APIs fine, but now you have a new problem. Discovery on APIs is an entirely different beast. 'cause now you're trying to beg somebody for a swagger doc or open API doc or some way to understand how that API is done. If they don't do Or, you know, GraphQL at least has its own, its own introspection, which helps. But if you don't have that, you're hitting an end point that tells you nothing. it gives you, gives you nothing to work from. Um, are you able to hear me now? Right. Sorry.

Chris Romeo 40:23

Yeah. Yeah, I can hear you. I think Robert's having some, some difficulties here.

Dan Küykendall 40:28

So if you're, you're If you're dealing with API, you have nothing to work from, right? You're, you're at a, at a ground level, you know, can't do anything. So, uh, when you're in that situation, uh, they're trying to, they're building o other stuff. They're building other tooling to help, uh, deal with discovery. And so that's what they're kind of focused on. And, um, and that's, you know, that's a fine problem. We tried to do both and we were doing more and more at Rapid7. Uh, But it was definitely an evolution. Trying to handle both is very difficult. We had at least a big enough team so we could, but, uh, but it's a tough challenge. Let me go back to another thing. I said, your use case was wrong. Imagine you're a different kind of user now. You're in a large organization, and this is a lot of our customers, okay? They're in bigger organizations. I talk to these IT guys and they, you know, the security guys and they have no idea how many apps they even have. They've got hundreds, thousands, in some cases, their difficulty is even just discovering them and then, but they're then need, they need to go and run a scan to see if there's any problem across the org. Right. What other tool can you point and shoot at all these environments without having to engage with every dev team and trying to like, I mean, that would be endless if you've got 2000 apps. Trying to engage with every team, getting a dev environment set up so you could run a test or run their static analysis, all that, like that's not gonna happen. So you could take that list, feed it into a DAST scanner, have it run all those thousands of scans over a month or whatever it's gonna be. You get some results, and if there's anything important, then you engage on a case by case basis. Right? That's a better use case for, you know, like then, then your single application that you're trying to get into a DevOps, you're, it's, it's the wrong tool for the job.

Chris Romeo 42:27

Yeah. Yeah, that makes sense.

Robert Hurlbut-2 42:28

Hey, Dan, let's, uh, take a look at what we have, uh, we've been doing for a few of these podcasts, a lightning round. So we have some questions to ask you. Uh, what is your most controversial take on application security?

Dan Küykendall 42:44

Um, well, what controversial, I don't know. I mean, you know, I think the, the the idea that they all, all the products suck was a fairly controversial take, you know, across the board, they all suck. Right. And, and I even included not just products, but services like pen testing and bug bounties. Right? And, and the good, the good and bad of 'em both. But I do think that on some level they all have a problem and. Um, and that that problem is not generally being honestly assessed and honestly talked about. Everybody wants to talk about why their solution is better than all the rest and, and not like being open about the, the strengths and weaknesses across the board. So I would say that's probably the most controversial 'cause it's created a lot of conversation and it's been, it's been a fun series.

Robert Hurlbut-2 43:35

I bet. So, uh, second question, what would it say if you could put a single message on a billboard ad inside of the RSA or Black Hat Conference?

Dan Küykendall 43:48

Man, I think that the legacy vendors should be watching their back because there is gonna be some innovation in this space. People are coming, there are a bunch of startups trying to solve this differently. Looking at it fresh and, um, and so if I were, if I had like some kind of billboard and I can express like, you know, have like a, a little crowd of legacy vendors and, and you know, they're getting old and they're getting ready to get shoved off a cliff because some of these startups are gonna be coming for 'em and, and they're not gonna be ready for it. And so I would say watch your back.

Robert Hurlbut-2 44:26

Cool. And then, uh, third question. What's the most recommended security book that you, you can give to us? Or recommend.

Dan Küykendall 44:38

Oh man. Um, oh gosh, what was it? Uh, was it David McGraw? His like, Well, one maybe not a security book, but I think far too many people have lost the art of, of understanding TCP/IP and, the, and really. So like, there's like the, was it TCP/IP Unleashed and then there's like McGraw's, like Software Fault Injection book. Uh, Gary McGraw there we, Gary McGraw. Um,

Robert Hurlbut-2 45:11

Gary.

Dan Küykendall 45:11

Gary McGraw's book on, and I don't remember the title. I mean, this, I wasn't prepared for this one. Uh, but, uh, there's a couple of those old school books that I still think are some of the best at really opening your eyes that to like how this stuff is all put together

Chris Romeo 45:29

Do you remember, uh, TCP/IP illustrated?

Dan Küykendall 45:32

That's the one that I said unleashed, but yeah, Illustrate. Yes.

Chris Romeo 45:35

the white. It had the white cover. Yeah. I had, it was two volumes. I think I've got it on my shelf in my closet.

Dan Küykendall 45:39

I, I have it in the garage. Yeah, exactly. That's the one I'm thinking of.

Chris Romeo 45:42

It's old though. That was like nineties, late nineties I think when that came out. Or maybe even a little earlier.

Dan Küykendall 45:48

But, uh, but people these days often don't understand how this is actually all put together, and I think they're missing fundamental, like building block knowledge and they're just jumping in on the top of layer seven and forgetting all of the other layers that actually make this happen and, and that those are also p part of the problem. So, uh, Yeah, I'm an upset guy, so I spend most of my time on layer seven, but, uh, I feel like it wouldn't be as effective if I didn't know the underpinnings of it and,

Chris Romeo 46:18

I'm with you as well. Like that's a thing like when, when I go to troubleshoot a network problem in the house, foundational knowledge of how TCP/IP works comes in handy. At different times when you're trying to figure out, is this a physical problem? Is it an addressing problem? Is it something else? And that's something that, uh, Yeah, I'm, I'm grateful that I learned that early in my career. I, I think I was forced to learn it early in my career by the people I was working with. They said, no, you have, you have to understand this. If you don't understand this, you can't even talk to us. So go away, figure it out, learn it, and then we can, then we can start talking about security when you understand how things talk to each other. So,

Dan Küykendall 46:54

Yeah, there's, there's much more beyond port 80 and 443 There really is.

Chris Romeo 46:59

Some people's minds are just blown. Right? It's like, what? What are you talking about?

Dan Küykendall 47:03

What's Port 80

Chris Romeo 47:04

Yeah. DNS doesn't use Port 80? All right. So Dan, from a, uh, kind of a key takeaway perspective, I. We always like our guests to leave our audience with something actionable. Like what do you, what what can people do as a result of all the things that we've talked about today, other than read, TCP/IP illustrated? I'll, I'll throw, take that one off the board, but what, uh, from your, your perspective, like what, what homework Do you wanna assign? What's your key takeaway for this audience?

Dan Küykendall 47:33

I think, uh, again, maybe I'll just go back to it and it's a broken record, but go back and look at all of the tools and start spending some time understanding the strengths and weaknesses. where these new vendors are trying to solve things a little different and, and they're gonna have their own strength and weak strengths and weaknesses. But I, I think just having that assessment, like, am I even using the right tools in my, for my problem? Go back and ask that question. Don't just assume like, oh, everybody uses a SAST, or everybody uses an SCA. Like, is that even the right tool for your challenge? And, and so I would, I would want everybody to go and assess that.

Chris Romeo 48:14

Very cool. Well, Dan, thanks for, uh, sharing your insights and, uh, your wisdom over a, a, a long career in the field of cybersecurity that kind of led into AppSec as well. And uh, it was great to hear you. Use all those terms that, uh, are stored in the back of my brain somewhere that, that get it brought out when we start talking about the past, like we did. But, and I enjoyed just getting your perspective on the OPSEC industry, the products. You gave me some things to think about with DAST. I might have to consider how I revise my, uh, my conclusion. 'cause I, you know, I think you, you gave me some stuff to think about though, about DAST in an enterprise is different than DAST in a startup, so most definitely. But we appreciate you being here and, uh, keep up the good work with your series that you've got going on and, uh, we'll continue to listen in as well. I.

Dan Küykendall 49:01

Thanks. It's been, been fantastic for me and, and thanks Robert and Chris for, for having me on.