Mukund Sarma -- Developer Tools that Solve Security Problems

Mukund Sarma, the Senior Director for Product Security at Chime, talks with Chris about his career path from being a software engineer to becoming a leader in application security. He explains how he focuses on building security tools that are easy for developers to use and stresses the importance of looking at application security as a part of the broader category of product security. Mukund highlights the role of collaboration over security mandates and the introduction of security scorecards for proactive risk management. He and Chris also discuss the strategic implementation of embedded security functions within development teams. Discover the potential of treating security as an enabling function for developers, fostering a culture of shared responsibility, and the innovative approaches Chime employs to secure its services with minimal friction for developers.

Links
Chime's Monocle
-- https://medium.com/life-at-chime/monocle-how-chime-creates-a-proactive-security-engineering-culture-part-1-dedd3846127f
-- https://medium.com/life-at-chime/mitigating-risky-pull-requests-with-monocle-risk-advisor-part-2-7013e1485bf2

Introduction to Overwatch
-- https://www.youtube.com/watch?v=QtZKBtw8VO4

Recommended Reading
Building Secure and Reliable Systems by Adkins, Beyer, Blankinship, Lewandowski, Oprea, Stubblefield -- https://www.oreilly.com/library/view/building-secure-and/9781492083115/
Drive by Daniel Pink -- https://www.danpink.com/books/drive/

FOLLOW OUR SOCIAL MEDIA:

➜Twitter: @AppSecPodcast
➜LinkedIn: The Application Security Podcast
➜YouTube: https://www.youtube.com/@ApplicationSecurityPodcast

Thanks for Listening!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Transcript

Chris Romeo: 0:00

Mukund Sarma is a seasoned security professional with expertise in application security, security architecture, and platform security. As the head of product security at Chime, he leads initiatives in AppSec, cloud and infrastructure security, security engineering, and embedded security programs. Mukund is also passionate about advising security startups in data security and identity and access management. He joins us to share insights on proactive security, engineering culture, and the security tooling that's contributed to Chime's success. With a deep understanding of product and application security, Mukund offers valuable perspectives on building effective security programs that can be applied in your company today. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I am co host of podcast, but today, flying the plane solo. I'm the CEO of Devici, also a general partner at Kerr Ventures, and I am excited today to have Mukund Sarma joining us. Uh, Mukund, we have to jump right directly into your security origin story because we don't believe in any warm up time or warm up questions on the AppSec podcast. It's all about getting people to your story as fast as possible. So, how'd you get into this world of security?

Mukund Sarma: 1:59

Yeah, thanks Chris for having me. It's my pleasure to be here. Um, yeah, I have been doing security for about almost 10 years now. Um, started as a software engineer working on the crypto side when crypto didn't mean currency. Um, it actually meant like building, uh, protocols for, uh, the government of India back then. Um, I was working on a few key sharing schemes with them, um, so I'm involved. I went to the US to do my research and my master's here, um, and then, uh, got into the space of application security, which I found it to be way more practical and pragmatic and, to be honest, a lot more fun. Uh, the chaos was something I enjoyed, uh, so, uh, started working on AppSec, um, started with consulting with Synopsys, um, wanted to go get a lot of exposure with a lot of projects initially, um, and that's kind of what I did for a while. So, Then I moved to Credit Karma, where I was there, um, one of the first few AppSec engineers, building out the entire program there with, um, and also eventually build, help build out the security architecture kind of program. Uh, been there for about five years. Um, and then, uh, I moved to Chime, been here for about three years. I am the senior director for product security, so everything security and engineering is what I work on. So, application security, infrastructure security, um, security engineering or platform security, some people call it. Um, and also a new function called the Embedded Security Program, um, which hopefully we can speak more.

Chris Romeo: 3:37

Yeah. Yeah. So did you, uh, start Chimes product security team? Was there a team there when you got there or were you one of the early people who was tasked with, Hey, put a team together here?

Mukund Sarma: 3:48

No, my boss, um, Jeff Trudeau, he was a CISO, he was here already and he had hired quite a few people. Um, and, um, Arkadi, who is the security architect here, he was running the, um, application security and, uh, infrastructure security folks. They were like what, they were about like five to, four to five people. Uh, but then when I joined, I called, we, uh, onboarded a new team called security engineering and then made that whole function as product security. And, uh, teams about 22 of us, including two managers, um, mostly mixed with a lot of software engineers, a few SRE engineers, and a few like security AppSec engineers. But primarily building like developer tools is kind of what the team does. They

Chris Romeo: 4:33

do you use the words AppSec and ProdSec? as synonyms of each other or do you define, do you think of them as different things? Because this is an interesting question and a lot of people are in our industry are answering it in different ways. Some people say, oh, they're different. Other people say they're exactly the same. So from your perspective, you're a practitioner who's doing this in the field. You use both of those terms at the same time. Do you, are they synonyms for you or are they different?

Mukund Sarma: 5:00

They are, and, uh, I think application security is a sub team under product security for us. So what I mean by that is we are building. Um, we're adding controls, we're adding tooling, we're doing, um, building a testing program, all of that stuff from the, um, application security side, but we're also building features that our members will use. We are also securing the infrastructure side of the house that our product is built on top of. So there are other components into the product security program, which, like the application security program, is like a subset of

Chris Romeo: 5:33

Okay. Yeah, I like that, that definition and that approach as product security and then AppSec sitting underneath it as a function because, to your point, like, is infrastructure security? AppSec? Well, not, it's not really an application. And for so many years we've bundled everything together and we've, we've called things applications and functions applications that really aren't. So great, great to get your take on that. Cause I know you're, you're living it. You're, you're, you're in the trenches kind of being a part of it now. I know that, uh, proactive security and engineering culture are things that I've read that, that you've described for what, what your approach is at Chime. And I'd love to hear more about what do you, what do you define as proactive security and engineering culture?

Mukund Sarma: 6:23

I think if I were to boil it down to one way, it would be collaboration. Um, and I think that is what, how I look at all of this. It's like, uh, we strive towards more collaboration instead of the best security, uh, function or anything in that sense. I think, uh, more often than not, what you see end up with most security teams is they burn the bridges between the various functions because they're trying to get out the best and the most secure product out there without. Actually realizing it's all part of the collaboration process of yours. Like you are working with the rest of the business to make sure that what they do, they understand the risks of it, if they're adding anything risky in there. And if they're not, at least they know who to reach out to if they want to like bounce ideas off and work things in there. So, um, I think, yeah, like the one, if I were to boil this down to one word, it would be collaboration, but to go a little more deeper, it would be like, Also looking after, um, not necessarily being the approver per se, because I don't think that will scale. The way we have taken our approach is like, we are here as consultants for you all, and if you do want to run ideas by us, we are here to help you. At the same time, there are enough and more guardrails to make sure that It doesn't get completely out of bound. So, um, it's kind of a good mix of both, I would say. Um, so we don't really have a concept of you must get every design approved by us, like there's no concept for that. So, if you look at our ticket flows and Like, even integrities of things, it just says that the design was reviewed, we don't say that the design was approved. Like, simple things that make a huge difference for us because, um, it's like, I don't want people to realize it's like we're just helping them build things securely, but it's their responsibility. Then when it becomes the accountability and responsibility falls on security, then it's, it's game over. Usually, um, accountability has to remain with the people who are building what they're building. Um.

Chris Romeo: 8:21

So I guess I want to ask kind of the, the other side of the question then from, from a governance perspective. How, how do you govern an environment that is, ha, ha, gives people the freedom and, and security is acting in from an advisory perspective? How, what, what's the, where does the rubber meet the road from a governance perspective? Cause I love the idea. That's for me, what you described has always been the perfect state. That's how things should be. But then the realist in me says, how do I govern the people that don't want, that say, or aren't doing what they're supposed to do? You know?

Mukund Sarma: 8:59

Yeah, yeah, and I think this is where I was talking about the other guardrails part of things is like we have built, um, platforms and services and frameworks for the rest of the developers that are building, will use our frameworks or libraries to do it the right way. Um, for example, we've. One project we just recently launched out is, uh, Service to Service Authentication and Authorization. It may sound very naive and basic and one on one, but, um, uh, we have hundreds of services that are deployed abroad, and, uh, each of them were doing it in their own way, in their own format, which made it hard for us to even audit and authorize, like, is the authorization even working? So, what we did was, um, we built this platform level solution where, uh, All the developers need to do is define one JSON config file to say what's the route, what's the method, and who are the allowed services, and then we take care of authentication and authorization for them. So all they spend is about five minutes, and we give enough and more instrumentation and observability to say, this is who all are calling you, do you actually want them to allow them to hit your endpoint, or is this a misconfiguration, things in that sense. So, um, that's another like a guardrail kind of thing. So now we have seen like people want to use our solution because it's the easiest to get out into the, um, to be a part of the ecosystem, uh, even if they couldn't do it.

Chris Romeo: 10:17

that's the Overwatch. Was that the overwatch?

Mukund Sarma: 10:21

The, no this is the service to service platform. Uh, Overwatch is our, uh, orchestration platform to look at how Um, to look at the code, um, and changes being introduced into the ecosystem. So for example, our, most of the time is GitOps driven, like I want to say 99 percent of it. So, um, depending on what the context is, okay, are you adding a new Ruby controller? Then we go scan a bunch of Ruby scanners and then leave inline comments to your GitHub pull request, if there are

Chris Romeo: 10:51

got it. I want to, I want to come back to that. But you, you introduced something I didn't even know about. So now I want to, I want to understand that better. So you, you called it, was it the Simple Services?

Mukund Sarma: 11:00

So it was called Service to Service Authentication and Authorization.

Chris Romeo: 11:03

Server? Okay, got it. So, here's the question I have for you. And, I'm going to give you a minute. I hear that and I think, okay, this, this

11:12

works at Chime because,

Chris Romeo: 11:14

and I think it's because you have a single, do you have a, is it a single product ultimately that you're delivering or are you able to do this across a multi, uh, collection of individual products?

Mukund Sarma: 11:26

Yeah, we can do it across multi. So we have. Uh, Services written in Ruby, Python, Golang, things deployed as Kubernetes, things deployed as Lambdas. So regardless of how you deploy, where you deploy, you can use this function.

Chris Romeo: 11:42

Okay, but it has to be, it's, so, so it's within the context of a single stack though.

Mukund Sarma: 11:49

Correct.

Chris Romeo: 11:50

So that's because I think, you know, when, when I hear about things like this, I always love to try to give folks that are kind of in different contexts, a perspective from your, from your opinion, like if I'm in a bigger company that say has 10 different product lines.

Mukund Sarma: 12:05

Yep.

Chris Romeo: 12:05

And could I use this same concept with, and maybe they're not all that, maybe they're not single stack. Maybe they're using different, maybe one of them's written in Java because it was older and something else has written in Ruby on Rails because they, they migrated to something that was modern and they wanted to, to use a modern framework or something like, how would you see that playing in and something that's not so, not, it's not the, it's not as, it's not a single kind of stack approach.

Mukund Sarma: 12:32

Yeah, so I think to clarify, we also have a similar problem. So we have things written in Ruby, we have things written in Golang. We have things written in Python. We have things written in the, uh, Node, Node ecosystem. So all of them still can use our solution because ours is at the platform level. So what happens is it's a sidecar deployed to every app. That sidecar is the one that's doing the authentication and authorization. So regardless of how, what you've written in your application, uh, you can still get in your authentication and authorization function in this case. Exactly.

Chris Romeo: 13:03

real benefit is authentication and authorization of a service has really become transparent for the developer because they don't have to do a lot of thinking. They just have to put a couple of lines in a JSON file and then your engine for authentication and authorization does all the rest for them.

Mukund Sarma: 13:25

Exactly.

Chris Romeo: 13:26

So I got to imagine they're loving it because you're making their lives easier. So is this part of the key to success though, is making developers lives easier versus adding friction?

Mukund Sarma: 13:36

Now, 100%. It's about building a lot of developer tools. I hate calling security tools. I keep calling it as developer tools. Building a lot of developer tools, building a lot of self service portals, um, all in the realm of adding enough and more guardrails. So, you do want to make sure that no one accidentally opens up something. At the same time, you make it easy enough for people to do it. For example, in the same e, in the same solution, um, there are ecosystem where a particular endpoint needs to be available on the edge, right? And, um, they, that means if it is available on the edge, that means it, the authentication and authorization is open. So the way they do it is they explicitly add a new field to the json file to saying that this edge access is equal to true, and then our platform will not break it for them. Whereas if they did have that flag set, then by default all the connection, all the connections to that endpoint is denied by default, unless and until they explicitly have that enabled. So this is where we're adding enough and more guardrails to the ecosystem to say that. You can still go do what you want to do as long as you're explicit about it and you know how to do it. This is how you do it. Yep.

Chris Romeo: 14:47

This is the first time that I think I've really understood guardrails. Like I've understood it theoretically, and I've understood the goal. And, but this is the first time I've actually had an example where I'm like, okay, this actually makes sense. And you're making developers lives easier because you're building a developer tool, not a security tool. It's a developer tool and no developer is going to say, no, I'm going to write my own authentication and authorization because I want to spend two weeks on this instead of, Five minutes making Mukund's JSON file that he's defined for me.

Mukund Sarma: 15:24

Yep. And also everything around it is like very low latency. We do it under one millisecond. So people don't even see it. They don't even realize that it's happening. We allow multiple different environments. So if you can have a different config for your test versus your dev versus your. Prod, non prod, all of these ecosystems can have different conflicts depending on how you want to set it up. So, uh, and also dashboards, a lot of dashboards, uh, your Datadog dashboard, observability, everything to say what's happening at every point.

Chris Romeo: 15:55

So what if I accidentally create a service that doesn't have a JSON config file? Do you just kick it out and it doesn't get to production?

Mukund Sarma: 16:03

By default, everything is, uh, locked down. So if they're, so every, we have something called a service launcher that we have built with our infra team. So we, like I said, we have hundreds of services. So every time someone wants to go create a new service, they go through the service launcher. And that service launcher has all your security defaults, pretty common practice nowadays. And by default, you get an empty JSON config with only a health endpoint, which is required for the platform to know if the service is healthy, but every other endpoint is not configured. So what it means is every other endpoint will not be taking traffic unless and until someone actually goes and edits that JSON file to have that.

Chris Romeo: 16:48

Okay. Nice. All right. Well, I've already learned a lot, but let's keep going. I know I've heard about and read about this Monocle tool. And so let's, let's go there next, because I want to, I want to understand what is Monocle. Okay.

Mukund Sarma: 17:03

Yeah. So, uh, it's a, it's a gamification platform where in order for us to reach out to every service owner and tell them, Hey, go do this X for security reasons, or go to X for compliance reasons. We've set them up as security score factors. Um, for context, when I first started, right, um, we didn't have, we were just getting started on this. So one of the first few things we had, it was does every. Repository have their main brands protected and has at least one reviewer, like it may sound stupid, but when you start off as a security team without, without, without that being a function previously, um, that is important, right? So you start off with that and then you say, okay, do you have, um, your SAST enabled? Do you have Dependabot enabled or whatever is your SCM tool? Like all of these could be scope factors that you keep having in there. And then soon, but as you mature your program, you can add more and more factors to it. Another thing is like now we say, are your critical and high findings patched with an SLA? Are you not running as a root user in your container? Are you, um, things in that sense? So all of these are scope factors now that we have kept adding as we mature our program. The, um, The trick over here is to make it simple enough, very bite sized work. Things that a developer would spend less than two hours on to get that task done. Because the minute you make it too hard, they lose the momentum, they lose that energy of wanting to go and do that particular task. So, What we have done is we have taken all the heavy lifting, created runbooks, created examples, guidelines for people to say this is how we do it, and then we make that into a score factor. So last year we migrated all our containers to run as non root, for example. So, um, the way we do that is we give them examples of how we do it in different tech tech stacks and how we, how we can achieve this. So we could get adding that as a score factor, made sure that everyone, um, went and did it because if they didn't, then the score would be bucked down. And people wanna stay on top of the leaderboard. So people wanna get their a, they wanna get their A plus, so they will keep making sure that their score fa their, uh, all the security requirements are met in that

Chris Romeo: 19:15

Yeah, John Doerr in the famous OKR book, Measure What Matters, like this is what you're doing. You're measuring what matters to the business from a security perspective. So, so many questions about the score, but I'll limit myself to just a few. So you mentioned SAST and SCA. Do you have a set of tools that are, do you have a stack of AppSec tools that influence my security score or do I have some freedom as a developer to use maybe Bandit for a Python, if I'm writing in Python, can I use Bandit as open source static analysis or do I have to use something you give me off the stack?

Mukund Sarma: 19:55

you can use anything you want, but we will always have something that we want. So you, you as a developer can go to whatever you want to, but at the same time, we will have a list of things that we will run and we will make sure you add the two. Um, and that is through the co factor part

Chris Romeo: 20:09

Okay, and you provide those though. You, as a developer, if I'm a developer, you're providing that stack to me, uh, as a set of tools that will influence my security score result.

Mukund Sarma: 20:20

Yes, and for example, that is what Overwatch does. It's like instead of people having to measure and keep track of every single thing and every single scanner to a language tech stack and make sure the rules are configured and which roles are enabled. And maintaining, basically, what we really wanted to solve with OWASP is not having to maintain hundreds and hundreds of CI configs all across various repos. And every time you have to change one thing, you have to go modify everything else. So, we created one GitHub app that runs as an orchestrator, and you can just maintain that configuration in that GitHub app. And that app is run on every single repository. And the context is handled at the runtime by the orchestrator here. So it's like, okay, you're Python, you can go run Bandit and here are the rules. Similarly, if you have something specific for yourself, you can enable those particular rules only for your repositories, for example. And we let developers maintain these if they want to, but we give them a standard set of defaults. But

Chris Romeo: 21:20

Okay, so let's, let's talk a little bit about gamification and the impact that it has here, because I know it's, it's becoming, it was a hot topic a few years ago. It kind of got a little cold, but it seems like it's warming up again as a, a topic here. And so how do you use gamification then? And you already gave us a little bit of a preview and you said people are driven to want that A and A plus on the scorecard, but tell us a little bit more about gamification.

Mukund Sarma: 21:45

yeah, initially it was just purely, let's, how do you get visibility out and how do you get people to know what's even matters? And, um, while we were talking to teams and building those relationships, this was good enough of a place to start with. You know, it's not too big. It's not too overwhelming. People know what exactly needs to be done. So they'll be like, most people, most engineering managers are like, yes, I want to help security. I want to do the right thing, but I'm going to spend only X amount of time every quarter or every sprint to do that. So tell me what I need to do. And then that is the minute most teams Don't do get this right is because at the point of telling me what I need to do They don't really have a good list of what needs to be done And that's what we get monocle to tell them is like this is how you do it and you're required to have at least a B or higher because if you don't have a B or higher that will go back to reporting and No one wants to seem look bad in front of their own bosses or things in that sense So it's out as a PR like let's report on it Um, we have also like, uh, played around with the idea of having it part of people's bonuses. Uh, luckily everyone has like, kept it, um, above their score so far that it's never actually hit anyone's bonus so far. But, um, yeah, that's another thing is like, how do you actually Drive some of the changes by having it part of like, um, the leadership's work system. So,

Chris Romeo: 23:03

Yeah, I mean, I think that's the ultimate gamification play, is money. Like, it's, some cultures support it, some organizational cultures that I've seen support it, and others don't. But, it is the ultimate driver of behavior. Like, if you attach it to bonus, everybody pays attention. And I've seen other organizations that have done that. And guess what? It gets a lot of attention because everybody's talking about it and all the senior leaders are asking people on their team, What's our score right now? Where are we? Because they're calculating their bonus at the end of the year or whatever as a result. So it's, it works.

Mukund Sarma: 23:38

yeah, we are also part of like the engineering monthly review. So, one of the factors that the security and seeing how everyone's scores are and who is below and this is part of the CTOs group. So it has a lot of visibility in there.

Chris Romeo: 23:52

Are there any negative challenges in scoring applications like this? I'm like, I'm just trying to think like there's, there's, if something is so good for a culture and so positive, there's always something that can go wrong with it though. And so from your perspective, is there, is there any negatives to this?

Mukund Sarma: 24:11

Yeah, I think so. I think, um, one which we early on realized was, um, if there were tasks that are bigger than let's get this done in under two hours, um, people are not going to like, they'll be like, it's not part of a monocle score. Like, why do we do it? Um, kind of a thing. And that's kind of where you have like, Talk to them and hope that that your relationship with them is like good enough for them to realize why you are doing this Or what does he ask? But also it's like showing them that you are willing to help by sitting with them, pairing with them Things in that sense that has helped a little but I think it becomes a very Here's the things of list of things I need to do But those are very tactical and not sometimes not everything is tactical you have to Sometimes rebuild something ground up, for example, or help the, like, we need to work with the platform team to help them, help them add security controls on them. And those are not going to be a two hour. Set of words. They're going to be pretty bigger. So how do you get some of that, uh, bigger chunks of work in? Because we treat Monocle as KTLO at this point. It's like just doing the right things to make sure you have the right minimum security score. But we want to go above that. We don't want to keep always at that minimum score. So that is one. The other one is there are going to be Services or attacks that are deprecated or old and just having to wait, wait to get that deleted. So no one's actually going and adding new features, but it still serves as a critical function. So how do you keep those minimum set of things done in there? Because, um, I can't say go rewrite this whole thing to use our new platform control because they're like, we don't want to rewrite it. So, um, how do you not?

Chris Romeo: 25:48

a big task, right? It's not like it's a two hour task. It might be a two month task.

Mukund Sarma: 25:53

Exactly. And maybe they don't even have the skill set for people who are there anymore. Like, all they know is how to keep this up and running. That's all they know about us right now. So, um, so then how do you, like, re like, there's a lot of exceptions to this policy. It's like, okay, depending on the context, we re score you. So, We also like the scoring mechanism is not just pure binary. It's a lot of context in there. It's like, okay, are you serving external endpoints? Are you treating member PII, things in that sense. And all those factors go into like how you, how your, uh, your service gets scored. So we can play around with those tweaks, but that's part of the gamification.

Chris Romeo: 26:28

Yeah. Okay. Hmm. Yeah. So I could see now, now I'm kind of seeing the light. Like you could be stuck in that just to summarize, you could be stuck in that situation where people are like, uh, that's not in the security score. So sorry, Mukund, can't help you. I'm focused on, I'm keeping my score at an A because I like getting an A on my test, but what you just described is not on the test, and uh, so add it to the test if you want me to, uh, to have to do it. So yeah, I guess that is a little bit of a, there's some push and pull in, in, in that type of, uh, of a challenge.

Mukund Sarma: 27:02

Yeah, also sorry one last thing is I think I would say if you don't like I think our gamification was hitting a challenge if it's just like keeping on top of the leaderboard right there's no other like that is good and exciting for a while but why else is it important like why else is it needed there so you need to like look at what are your other ways of Um, adding to the whole gamification process and I think for us it was a bonus part, it was like, a percentage of the bonus is tied to it, but if you don't, if you can't find that, then your gamification dies, that spark of gamification dies quickly, so you'll have to find newer ways to keep that in,

Chris Romeo: 27:35

It has to be a motivational lever in the equation, right? That, that causes people to have to pay more attention to it and, and quarterly can circle back around because they're looking at how much their bonus is going to be impacted by it. So, Just to help people kind of close their thinking on the security score here, uh, if you could throw out just a couple of examples of things that go into the score. We don't, I know there's a lot of things, we don't need to go through all of them, but I just want people, I don't want people to leave this conversation and go, eh, I kind of think I knew what they were talking about, but I didn't really grasp what types of things you would be scoring.

Mukund Sarma: 28:14

Yeah, we've actually written two good blog posts on it, we can drop it in the, uh, description here, but I think of, like, let me talk about two, three of them. One is like, um, are all vulnerabilities for this service patched within DSLA? are critical and high vulnerabilities. As long as they're patched within the SLA or they're still within the SLA, they get a full hundred points for that particular factor. Second factor could be, is the service running as a non root in their docker container? Third one is, um, have they patched their, uh, image in the last two weeks or in the last one month or whatever frequency that you define? Um, it could all, it could just mean as much as. Um, using our base golden images and then them having to just redeploy once in every two weeks or once in a month. Most people do that often.

Chris Romeo: 29:05

Yeah. So they get the changes out of the golden image that you've already, you've already done the work for them with the base images as far as patching.

Mukund Sarma: 29:12

Yeah. So these are like some three things. Um, we can go more and more. Like I said, as you mature your program, you can add a lot of more factors and remove factors that don't make any more sense. Um, one of the factor we spoke about is the services service auth is, is the service using it. So that is another way of actually getting them to use it because it's part of the score factor.

Chris Romeo: 29:31

Yeah, but over time, you'll reach a point where everybody will be using it, new services will get it by default because people, or maybe that's, it's, they're forced to add it by default. And so I guess you reach a point where everybody gets a hundred points. And so maybe that's a good measure for when do we retire things from the score is if everybody gets a, gets a hundred and then we should pull it out of the score.

Mukund Sarma: 29:54

exactly. Yeah. That's how we do it today. Yeah,

Chris Romeo: 29:58

very cool. So let's, let's transition and talk about embedded security function because I, I've, I'm curious about that. And that's where it feels like we've been in the technical side, in the tooling, in the culture. I think the embedded security function kind of is going to bring us to the people side, kind of melding together with the culture. So, um, a lot of attention on security champions these days, but let's, so let's give us a definition when you say embedded security function, what, give us a definition of that as a start.

Mukund Sarma: 30:33

we have different domain engineers, part of the security team, but working with the different domain teams. So what I mean is I do have a data security function at Chime. So what that means is I've hired a data engineer. Who actually didn't really have a lot of security, um, experience, but we hired her. She's part of my team, like she reports to me, uh, but is working with the data org to make sure the data security function is built up and it is pragmatic and it makes sense. And, um, it actually, um, adds value to their, um, development lifecycle. So, um, that's one example. I think more often than not, uh, it's easier for us to teach. Other domain engineers or security concepts than the other way around. Um, I'm not going to have a lot of luck trying to get someone, a security engineer and prioritize them how, um, data engineering or data science concepts are done and how is Kafka built and deployed or how is streaming done and how is, um, Your ML models, built and deployed, all that kind of stuff. Whereas if I do hire someone who knew all that, that is their domain knowledge, and taught them the security concepts while I'm supporting them or the team is supporting them with that, it's much, much more effective in this case. Um, one because more often than not I've seen is like when we, if it was the other case where we did hire a security engineer, Um, the recommendations are not practical or pragmatic. It doesn't make sense for the data team to even be like, yeah, that doesn't make sense for us. That doesn't make sense when you're building like Java services and deploying it. But it doesn't make sense for us. Like, that's not what we do. We are building a platform here. Um, you don't understand how ETL is done, or you don't understand how transformation jobs are written. That's kind of the feedback we've always gotten. So what I've done is we've hired domain engineers. Then, they report to security, but they are part of the data team in this case. So they take part in their stand ups, they take part in their demo as, they take part in all of those things. But, They report back to me over here. That's the concept of an average.

Chris Romeo: 32:43

How did you choose the order? of embedded security people to add. Cause I could see, uh, uh, uh, organizational problem. There's, we never have enough resources, right? Even on security teams. We never, it's not like we have an unlimited, we can just hire one of, of every department to be here. So how did you prioritize, this is the first person I need. This is the second person. And then these are the people that I'll be able to add in the next coming years.

Mukund Sarma: 33:09

Everything is risk based. Everything we do over here is risk based. You'd want to see what is the biggest risk for the company, what is the biggest risk for the program, and then try to see how do we try to attempt for it, and what's the best ROI here. Um, for example, the data one, I think we realized that we needed that because, one, we didn't have that expertise in the security team, so we were actually burning bridges every time we tried to have a conversation with them because They are not talking the same language and, um, we really, they actually also need the help to understand what it is and also probably even help. Pair with people on the data team to do the job because it's not an easy enough thing. Um, I think as an AppSec or a whole security team in general, as an industry, what we have, we have tried to handle things purely from the AppSec point of view. It's like, okay, if people are building features, if people are building services, um, then we have enough and more like education, tooling, resources, all of that stuff in there. Right. We have not done justice for the data ecosystem, for example, like we don't really treat data engineers or data scientists as first class engineers for the most part. That's what is the reality in most teams. So there are no good OWASP top 10 or there is no good, um, tools to get a model deployed and build periodically or things in that sense. So I feel like, um, we need to like reinvent some of those. Paradigms and the other domain knowledge. So, to answer your question, I think it's purely risk based. So, uh, for us, we thought this is what made most sense at this point of time. So, we hired that person. Um, I do have another person on the part of the data embedded security program that's part of a different platform team that we are building, um, at Chime and, um, they are trying to learn and help the whole money transfer part of things done securely. So that's a whole different data, it's a whole different domain where they need to know the integrities of how a processor works, how is the, um, how, how all the money transfer aspects work and then help them secure it. So it's a whole different domain, it's a whole different site.

Chris Romeo: 35:17

Mm hmm. So if a, let's say somebody listening here is thinking to themselves, this embedded security function, this sounds like a good idea. I want to do this inside of my, my company. What would be some advice you would give them for somebody new who's trying to do, take this thing, you, this concept you've already done, had some success with, what would be your advice to them on how to, how to successfully move forward?

Mukund Sarma: 35:42

Make sure the domain team wants it. Otherwise, don't do it. That is the main thing, right? You have to talk to the leadership there. You have to talk to the team there. You should make sure there is appetite to even start some of this work in that whole problem of space and then start looking at it. Um, the other thing you want to do is not challenge their, um, uh, people, in that case, for example, uh, you want to make sure that there's no, like, a shadow data team, even in security, that's trying to do their job, right? We all hate shadow security teams, shadow IT teams. Similarly, even those people hate shadow data teams that are people who are trying to go to a data, in this case, for example. So you need to make sure that relationship, that understanding is there that, You are just trying to help them, but it's really that they are the data domain experts and, or they're the domain experts in this case, and then that they, we will be working with them to make sure that's done the right way. Um, people who start off without that can really like, this can backfire really well, like really quickly, um, if they don't pay attention to that aspect. Um, the other one is, um, do you have the appetite is also important is like, because I think what you realize is like, There's a lot of hand holding, there's a lot of work that needs to go in. And does you and your team have the appetite to do that extra work? Because, um, like I said, most of these, uh, most of these domains don't, have not really had that level of engagement with the security team or, with, I think in that sense. So, uh, you'll have to find paradigms of, Things that are normal to you, but explain that to them and work it out. So it's a lot of work on your end too. So I would say those two. Um, and definitely the person who you hire, right? Um, if it's a more junior person who hasn't had experience doing, um, even working in a normal security function, forget the cross collaboration, then, um, you're answering them for them. So successes. So just be sure and kind of people you hire, like,

Chris Romeo: 37:41

Yeah, some, some more skill sets on the cross collaboration front seem like that would be, which kind of leans itself towards more of a mid level experience to a senior person because they're going to be, especially in the early days, they're going to be front and center. In front of that team, like there's going to be a lot of, a lot of things are going to be determined in the first month or two. When the team's trying to feel that person out, like, Oh, is this person from security just here to look over our shoulder and, and report back on all our failures? Is that why they're here? Or there's that there's building up that trust where the, the, the data team in this case feels like, okay. This person is here to help us. They're not, they're not trying to find our flaws. They're trying to help us secure our data better. They're not telling us what to do about the stuff we're experts in. They are only offering suggestions based on their security knowledge and how it applies to our domain. So I can see that being one of the key things you got to watch out for. Cause you could, if you get a bad relationship going in the beginning, forget about it. They're, they're going to, They're going to cut you out of meetings. Like, Oh, we forgot to invite you. Sorry. Yeah. So what, what, what do you see as the future state of this, this embedded security function? Is this something where the security team of the future, let's say the ProdSec team of the future, half of the people are distributed security are embedded security people where, um, your team is really just a whole lot of dotted lines. Is that what you see as the future state, or where do you see it going?

Mukund Sarma: 39:20

Yeah, I don't think so. I think, um, the, there's definitely work for some of these domains for the next like two to three years, but then I think once they have embraced this, once they know how the function, like how to pair with security, how do we work together, um, there might actually not be a lot of value for that, uh, particular person to be that embedded into that model anymore. So they can come back and work with the security team. They can work with us to build more toolings, functions, things in that sense. They can also be part of a different embedded program, a different team in this case, or something in that sense. So, but I also feel like you want to make sure that two things. One is, um, that person doesn't become the person to do all the security vulnerability patches or things in that sense. You have to be very careful of that to say that, no, my role up here is an advisor as a consultant. I will pair with you. I will work with you, but you're the one going to finish it up, um, if needed. Like, right. And, um, the other one is being in an embedded team is actually very lonely. Um, and you should be careful of that as a, like, a manager or someone who's trying to lead that function as well. Because, um, the rest of the security team has people to work with and talk to and things in that sense. Um, whereas this person can feel they're neither part of this team nor the domain team. So you have to make sure that there is room for, like, this person to grow and engage and be part of a bigger function too. So, um, you want to make sure that that is, that is also a very tricky part. Like, if you don't get it right, you can feel very, um, Um, Lonely as a part of an Embedded Engineer. Um, and also they simplify the process sometimes. For example, you don't want to have them go through two sprints planning, or two retros, or two different things, because they're part of two different teams. So, you've got to simplify some of those aspects for them. Like, those are more nitty gritties, but they make actually a huge difference for them.

Chris Romeo: 41:13

Yeah, that's, I love to hear you describe that because that just shows you're an empathetic leader who's thinking about your people. And that's a trait I wish we saw more of. In all industries of people that, you know, are thinking about, uh, how are we going to, how am I going to hire the best people, but also how am I going to have them here five years from now and just as excited about what they're doing in five years as they were the day they walked into the building. And that's, but, but your description there of, of considering and thinking about their experience and everything is just, it's a powerful message because It just shows your, you, I'm going to guess people like working on your team. I'm just going to make a guess. But if that's how you're thinking, the best teams I've been on have been in my life and my whole career have been where the managers actually cared about the people and they get out of the way and let them do what they're good at, which just sounds like the two, these are two things that your, the teams you build are doing those things as well.

Mukund Sarma: 42:11

Thank you.

Chris Romeo: 42:12

All right, let's do some lightning round questions. I could, we could, we could literally, I could spend the whole day talking about the things we've talked about already. But. You know, we got to summarize it. We're going to, we're going to link some blog posts where people can get more information about the nitty gritty details of Monocle and some of the other things you've done at Chime. Uh, but let's, let's, let's do some lightning round questions and have some fun with these too. So, first lightning round question is, what is your most controversial opinion on application security? And you have to say why you hold this view.

Mukund Sarma: 42:46

Um, I think regular software engineers, for example, are way better AppSec engineers than AppSec engineers, uh, because I would say that why do I hold this view? I think they know their domain much better and they know their aspect much better and it's so much easier to teach people the security concepts than the other way around.

Chris Romeo: 43:06

Yeah. Yeah.

Mukund Sarma: 43:07

I know a lot of people will hate me for that, but that's my controversial.

Chris Romeo: 43:11

you can reach Mukund at P. O. Box 102, Sioux Falls, Idaho. I don't know what it is. Don't, don't send anything there. No, I think, you know, and that's, that's, I think that's a fair assessment. I think if people have experienced leading teams that have seen people come from different trajectories. I think they agree. I like, I've seen people coming from multiple trajectories and I agree. I would rather have a developer and then layer AppSec on top of what they're doing versus going the other way around because it's, and it's, it's, yeah, it's not that controversial because I've, I've been yelling at people like everybody should learn. If you're in security and you don't know how to, you don't have a single coding language, you don't have knowledge of a single coding language, and then you're in AppSec. No, you, I'm sorry. You can't, you can't say I'm an AppSec engineer, developers, I want to help you. Uh, that language you code in, it's not really for me. I don't really, I don't know, I don't know what, sorry. But do this, and do that, because I said so. All right, now I'm getting into my controversial take. I took over your segment there, I'm sorry. How about, uh, if you could have a billboard message at the RSA conference or Black Hat conference that had like a single message on it, what, what would that message say?

Mukund Sarma: 44:24

collaboration greater than security. Um,

Chris Romeo: 44:28

sticker right there for a laptop sticker. Okay. How about recommended reading any books that you like to recommend to people? Any favorites, uh, in could be technical or non technical doesn't matter. I've read it. Great book.

Mukund Sarma: 44:41

maybe the, uh, building secure and reliable systems by Google, the, um, I think that's a very pragmatic, practical book and makes you really understand how complex systems are built and applied. I think that's a really good book. Um, non technical, um, I would say I'm more of a non fiction kind of person. So, um, right now I'm reading this book called Drive by Daniel Pink that talks about, yeah, that's a good one.

Chris Romeo: 45:12

What do you want to leave our audience with? Do you want to give them homework via call to action, or do you want to just want to, how do you, how do you want to end the conversation?

Mukund Sarma: 45:20

Yeah. I would just say like people to like think more from the developer's perspective, think more of what, um, how can you enable people to do this right. And, uh, try to relate this to how you can get. Your engineering or developer function to be more accountable for security and not you be the driver or the owner because the minute you are the one responsible for it, you just can't scale. Usually it's a 1 to 150, 1 to 200 list of people like you want security engineer for every 200 engineers and you just cannot scale that way. So I would say think more from how can you enable them to do this, right, instead of you being the gatekeeper. Yeah.

Chris Romeo: 46:08

Very cool. Well, thanks Mukund. Thank you for sharing your wisdom and insight into building teams, building programs, building tools. This has been a really excellent experience for me. And so we look forward to a future conversation in some months where we can dive into something else that you're, uh, you're focused in on. So thanks for being a part of the Application Security Podcast.

Mukund Sarma: 46:30

Thanks, Chris. Pleasure's mine. Thank you.