François Proulx -- Actionable Software Supply Chain Security

Software supply chain -- how deep does the problem go? François is here to help us realize how deep the rabbit hole of the supply chain is and enlighten us with strategies to get out of the hole.

François emphasizes the importance of branch protection in source code repositories as the cornerstone of any supply chain, highlighting the need for peer review and static code analysis before merging. He also discusses the concept of tag protection, which prevents anyone with rewrite access to the repository from modifying a tag. This is particularly important in the context of build systems, where an overwritten tag could compromise the entire system.

The conversation then shifts to a "Let's Encrypt" equivalent for package signing, which François believes is being addressed by the SIG store project. This project introduces the concept of keyless signatures, which eliminates the need to manage private keys, a process that can be risky and cumbersome.

François also discusses the importance of understanding your dependency tree and using package manager lock files to ensure that the version of a package you're downloading is the one you expect. He mentions the Terraform modules, where the lack of a lock file for modules can lead to security vulnerabilities.

Toward the end of the episode, François recommends listeners explore the OpenSSF (Open Source Security Foundation) and its various projects, such as the Scorecard project, which provides a security posture for your repo. He also mentions https://deps.dev, a free Google service that scans open-source repos and runs the Scorecard on those projects.

Look up towards the light if you find yourself at the bottom of the rabbit hole.

FOLLOW OUR SOCIAL MEDIA:

➜Twitter: @AppSecPodcast
➜LinkedIn: The Application Security Podcast
➜YouTube: https://www.youtube.com/@ApplicationSecurityPodcast

Thanks for Listening!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Transcript

S10E13 -- François Proulx -- Broken Links - Behind the Scenes of Supply Chain Breaches

[00:00:00] Chris Romeo: Francois is a senior product security engineer for Boost Security, where he leads the supply chain research team with over 10 years of experience in building AppSec programs for large corporations such as Intel and small startups. He's been in the heat of the action as the DevSecOps movement took shape.

[00:00:18] Francois is one of the founders of North Sec. And was a challenge designer for the North Sec ctf. Francois joins us to talk about supply chain attacks, some of the new things that are happening. We get into salsa, we talk about attack trees, and some work he did looking at source code control systems with attack trees, and really just provide you some actionable results and things you can take away about how to properly secure your software supply chain.

[00:00:48] We hope you enjoy this conversation with. Francois Proulx.

[00:00:53] Hey folks. Welcome to another episode of the Application Security podcast. This is Chris Romeo, CEO of Kerr Ventures, and also AppSec Aficionado, giving myself a new tagline here. I'm trying it out. I'll see if it works. I don't know if I really love it yet, but I'm going with it just cause I made it up on the fly.

[00:01:55] Joined by my good friend, 

[00:01:57] Robert. Hey 

[00:01:58] Robert. How are you? 

[00:01:59] Robert Hurlbut: Hey 

[00:01:59] Chris. Doing well. Yeah.

[00:02:01] Robert Hurlbert. I am a principal application security architect at Acquia. Also focused on threat modeling and uh, always good to be here to talk again about application security.

[00:02:13] Chris Romeo: Now, do you consider yourself 

[00:02:14] Robert an AppSec aficionado? 

[00:02:16] Robert Hurlbut: do, I do definitely. 

[00:02:18] Chris Romeo: I'm th okay, well I was gonna trademark it, but I guess I really don't need to now, cuz apparently other people think this too. So, um, super excited today to have Francois Proulx who's going to be answering some questions and helping us to understand software supply chain and the different attacks that come into it.

[00:02:35] And so this is based on a talk that he's done at a number of conferences already, broken links behind. The scenes of supply chain breaches. And so you can, you can actually Google that and find it. I, I watched a, a version of it from BSides nyc, uh, just earlier today preparing for the interview. But Francois, we'd like to just dive right in to the deep end of AppSec, and we wanna know your origin story.

[00:03:01] How'd you get into AppSec?

[00:03:03] François Proulx: Sure. I glad, glad to be here with you guys. Um, yeah, so, all right. So I guess it,

[00:03:10] things get, um, start, start to get interesting when, uh, I was about five years old, so, you know, I had the, like many kids back in the eighties. I had one of those Radio Shack computer, which had the basic programming language built in. And like many kids, you know, at the time were all retyping games that would kind of find in books and magazines and whatnot. Um, There was also like a pretty cool feature, which I remember in, in those Radio Shack computers, there was a, a basic, uh, statement called Motor on Motor off, which was originally intended to trigger the cassette tape, but it was really like a way to trigger any kind of electronic. Like kind of a, you know, on enough, uh, thing of, so would plug like at a relay and build all sorts of Lego contraptions and trigger motors that would plug into, uh, even VCRs to create a multimedia experience with that. So that's kind of how it started. And then later in middle school, um, I was spending most, you know, all my free time at, at lunchtime, at the computer lab with other kids that did that, uh, when they were young, like, like me. And one of those kid is, uh, il which many of you uh know. Um, and you know, it was the time of dial up internet with Netscape 0.9 and Mosaic and things like that. Um, and you know, I didn't have internet at home, so I would. Save as, um, webpages onto floppy discs. Bring them back home to teach myself html and later JavaScript by reverse engineering the, the webpages. Um, so with John Atan back then, we're again tinkering with writing our own text based adventure games and basic, and, you know, also kinda get a glimpse at social engineering.

[00:04:57] With him, he was, he's about two years older than me, so he would kind of, uh, teach me some, some, some tricks. Um, that he, he had, uh, already and later, a couple of years later, I got my first summer job as a technician, computer technician for the local school DIS district, um, which was pretty awesome as a first summer job.

[00:05:18] You know, building PCs, uh, passing ethernet cables in the wall and installing windows NT four or even 3.51 servers. It's actually a pretty cool first summer job well paid. But you know, soon enough I got my first lesson because, One of the thing I did, uh, the year before that was, Use those social engineering skills combined with programming to create a fake Windows OS login screen and trick the teacher into giving, uh, his admin password. I don't even remember what we ended up using the admin password for, uh, honestly, but, you know, we were in those, you know, phone freaking reading zines type of, uh, scene back then. So I guess someone snitched me because my boss at the school district heard about this thing that happened with weird os SL and screen, and thought that out of all the kids in the school, I might know something about that. And, uh, you know, in the end, I, I, I got the lesson by not getting, uh, renewed the next summer because, uh, I, I, I confessed to my my what, what I did. And he told my dad at the time that I needed to decide if I wanted to be on the dark side or the good side going forward. So that's how it started. And, you know, after an engineering school did a bunch of CTFs to practice, uh, put to good use those, uh, security skills. Uh, and later founded North Sec, uh, here a security conference and CTF in Montreal, uh, was participating in different, uh, CTFs, uh, like Defcon qualifications and whatnot. But really my first. Job, like real job out of the university was a, as a iPhone developer, cuz I was even tinkering with the jailbreak OS before it was official, like, uh, way to, to develop on, on iPhone. So I knew how to develop iPhone applications before it became official. So it kind of had a head start in, in the game, but always all those, you know, knowhow of SQL injection success and whatnot, uh, I was the one around the every dev team kind of telling the, uh, the, the, the developers to be careful about that.

[00:07:29] I was kind of always the annoying guy on the team. Um, back then, you know, there's no GitHub, so no pull review, pull request, but we're still doing some kind of, uh, code review. But, um, anyway, so I got, uh, not a couple of years later, I got my first full-time security job, um, in a startup. Uh, I was kind of, you know, literally kind of building a application security program from, from zero, just as like my first full-time security job, like in a startup of about 30.

[00:08:02] So employees, six months later, we were acquired by Intel. So pretty, uh, fast, uh, turnaround of like building that AppSec program and then joining Intel and then worked with Brooke Shenfield, which I think you, you guys know. So he was kind of my, you know, mentor at Intel. Uh, worked, uh, pretty closely with him as a product security champion. And then, uh, at some point they, they closed the office. I did a couple of years in the blockchain stuff, which don't really love, but, um, found my, uh, home at, uh, boost Security now where I am se senior product security engineer. So, so helping both on the security research side as well as helping define what the, the product, uh, and service that we're, we're, we're.

[00:08:48] BU building now is really, um, kinda AppSec, uh, orchestration platform that kind of aggregates results from all sorts of scanners and orchestrates. What, what happens with, uh, those findings gonna automatically to triage and, um, and, you know, and all that.

[00:09:07] Chris Romeo: Very cool. I always, I always love it when someone's story goes back to early childhood. A a and then maps forward. I love to hear those stories cuz it's so cool to, to to see the, the connections and the intersections and, you know, growing up with Jonathan Marcel and then Brooke, who is Brooke Shaunfeld, who is one of my favorite people on earth to, and, and somebody who I, I, I certainly cherish the time that I get to spend with him, which is not, not very much, but to, to be mentored by him is like, you know, to me he's he's the one of the giants in the industry, even though he's gonna be so mad when he hears that hears me, describe him as that, but he is.

[00:09:44] François Proulx: he taught me, uh, threat modeling and so that's, you know, been a great mentor.

[00:09:50] Robert Hurlbut: Excellent. Yeah, agreed. Agree about Brooke and, and, and hearing the stories of Jonathan as well. Uh, that's fantastic. So Francois, uh, you know, as we dive into, uh, the topic today, uh, could you briefly explain the, uh, this idea or this concept of supply chain attack, uh, for our listeners who may not be familiar, uh,

[00:10:13] François Proulx: Sure. So you know. I think, I mean, all your listeners are very much into application security, which, um, is sort of a, I would say, narrow focused aspect of what supply chain. If you kinda step back a little bit and just not focus on like looking at the source code, finding SQL injections and all sorts of os top 10 vulnerabilities by, you know, scanning the code and whatnot.

[00:10:41] If you kinda step back a little bit, And look at it, not just from the perspective, okay, there's a piece of code, it's gonna be running in production. I wanna prevent vulnerabilities from being exploited. But actually, if you're like, okay, well this place where the source code is actually stored, It is something that you want to care about in terms of security, just like you know, information security 1 0 1, like data security. It's not just the application, but hey, the source code, if it gets compromised, if it gets modified by someone unauthorized. Uh, either, you know, some outsider or insider, so like malicious insider type of, uh, threat. Um, so that's, you know, the supply chain is like when you start to step, step out of it and not just look at application security, but really the developers that are touching that code and where does that code go?

[00:11:33] And like it goes into the build system. And that build system cannot be compromised and then actually affect. The downstream code, like the what runs in production. So there are many, many areas where the supply chain, uh, comes in, like, uh, through the ways of your dependencies you pull in. Like typically nowadays, you're gonna have, you know, maven dependencies, npm pipeline and whatnot, kind of pull all those things from open source packages. So again, you kind of the, when you go into the, the down the rabbit hole of, okay, I looked at my supply chain, but then I step back and look at the supply chain of my, uh, own, um, dependencies. It gets quite complex. So that's the big picture.

[00:12:21] Chris Romeo: So when I think about the complexity of the modern software supply chain, it amazes me every time an application runs. It actually does something. It's like there, there are so many transitive dependencies that just buff it out across so many different projects and things that are coming together and then boom, it all comes together and it starts and it just works.

[00:12:46] And I'm like, it's just amazing to me that that much complexity can all come together and still work. So from the C I C D pipeline perspective, we, we, we know, you know, we did a, uh, an episode last year with, uh, Daniel and, uh, I can't remember the other gentleman's name who created the top 10 oas, top 10 list for C I C D.

[00:13:09] So certainly the, this area has gotten a lot more attention recently. Um, from your perspective, like why are C I C D pipelines the prime target? Why are they getting so much attention these days?

[00:13:22] François Proulx: I think in recent years. I mean, uh, you know, it's, it's in the news, it's in the, it is in the ether. You know, people talk about it much more. I think, you know, you know, it comes back to the talk I did, which is, you know, behind the scenes of supply chain breaches. I started to talk by saying that, you know, it's, it's what people talk about these days.

[00:13:42] It's, it's a hot topic in the news with SolarWinds. Like, you know, many people heard the, the term supply chain in the context of software engineering. Uh, Uh, related to SolarWinds, which we can get into, uh, a bit more. But, um, I think, you know, it's really been the kind of secret sauce of many nation state actors for much, much longer. So, like decades. Cause, you know, supply chain tricks, it's been the defacto thing of. Spies, kind of, you know, infiltrating organizations and replacing bits, you know, like changing the package, the content of the package, uh, on its way to the destination. Just apply that to your open source dependency, you know, you can look at, at that.

[00:14:31] So, uh, looking at the um, CI environment, it's really where it's supposed to come altogether. You're supposed to have the legitimate. Uh, source code that you built, and then you pull all sorts of dependencies. So this environment where it gets compiled or packaged and hopefully signed so that later downstream you can assert that it really came from your trustworthy CI environment, uh, that that's kind of a high security. Piece of the supply chain. If you don't have any signature, well then you cannot even trust, uh, that. But, um, I think, yeah, it, it is just like something that came in the news, like in the spotlight with those high profile breaches, but, It's, it's been the, the, the tricks of, you know, nation state. I, I think it's really because we're getting better at AppSec overall.

[00:15:24] Like, you know, with thanks to OASP and other initiatives in your podcast, people start to know the, the, the, the basics of good security hygiene. And now we're just moving on to the other, uh, weakest links. You know, like you might have a perfect zero vulnerability code base. But then you have a vulnerable pipeline with GitHub action that can compromise by accepting, you know, open source contributions. God knows what you know.

[00:15:57] Robert Hurlbut: So you mentioned. Mentioned, uh, SolarWinds breach, and I know you mentioned that in your talk as well. Could you give us a brief overview of what happened in that particular case? And I think there's some new stuff that's come out recently and we talk about that as well.

[00:16:09] François Proulx: Sure. Actually there was a, an article published by Wired, uh, beginning of May. So you may wanna look, look it up. It's called the untold Story of the boldest supply Chain Hack, ever pretty cool title. Um, so it goes a little bit further than what had been disclosed before. Um, and really solar wins is a, um, American company that's, uh, building a software called Orion.

[00:16:35] Amongst other things that is actually pretty popular amongst the us. Um, uh, government departments including US, department of Defense, uh, pretty high security environments where this, uh, network monitoring and management tool, uh, is, is used. Um, and, uh, one way that nation state actors found to infiltrate those high security networks was actually, uh, Through this supplier, this, this company building this software. And, uh, they in this case did not modify the source code directly. Uh, they somehow found a vulnerability into, uh, or found a way to get into the, uh, CI environment and modify, add A A D L L if, if I remember, it's in the article. That would modify at runtime in the build environment, which I think, uh, if it was a, a team city, it's like a, a JetBrains, I think a CI environment.

[00:17:35] So it would kind of modify the, uh, source code that would get built. So if you looked at the file system, you would not necessarily necessarily see the malicious, uh, uh, source code. It would get compiled and then, you know, downstream, um, uh, users of SolarWinds would, would get a tainted copy. Uh, so someone doing forensics, which was done by, uh, Manian at the time. Uh, and Microsoft's the, the, the, the forensics, uh, the article actually goes into the disclosing, which was not known before. That even the, the Justice Department was actually probably one of the first target, and they omitted to men to tell it to the F B I and the f b I. Later in the N S a, learned about the fact that the Justice Department was the first target, uh, about nine months or so before. It got into d o d, um, systems. So that was kind of the entry point into the US and they kind of tested, perfected their method. Uh, and, and, uh, the article kind of, uh, tells the story of those, uh, f FBI agents that were really mad to hear on a phone call that the, the, the Justice Depart withheld, uh, uh, at security incident that happened.

[00:18:51] Uh, that was exactly the one that they, they found out, uh, leaked further.

[00:19:00] Chris Romeo: Okay, so what are some of the lessons that we should learn from this? Like, are there, are there things that companies should be doing differently? As a result of, especially with the new information that's come out from SolarWinds, like I didn't, I didn't actually know that I, I, I glanced at that article. I didn't read it in depth enough.

[00:19:20] I should have read it closer, but I didn't realize the DLL connection, like, that's brilliant to modify a d DLL that you could, you, even if you scanned the text fields of all of the things on that hard drive, you wouldn't have seen any. It wasn't a script or something that I could find an artifact that was doing some action.

[00:19:39] It's buried in the D L L, which nobody's gonna forensically find, at least on a, on a quick pass of looking at it. So are there any other lessons learned that you would say we could apply to modern companies?

[00:19:51] François Proulx: Yeah, I mean, I think the, the, the. Holy Grail of, uh, of supply chain

[00:19:57] best practices nowadays is, uh, through the, the, the SALSA initiative, the S L S A, uh, which stands for, you know, those acronyms. Sometimes they. Have fun with it. Uh, I think it's supply chain levels for software artifacts. Uh, a bit of a mouthful, but, uh, that's really where I, I guess, someone who's interested in hardening their supply chain throughout and understanding the threat model of what you should. Pay attention to at the different stages, like the, the source where your, your source, uh, is stored source code, like source control, uh, management, like say GitHub, uh, your CI environment, which you know, could be GitHub, action Circle ci, or other kind of Jenkins server that you may be, uh, running yourself. And then where you store the artifacts, maybe your Docker registry or, uh, you publish, uh, NPM packages and then later you consume it in a. I guess Kubernetes cluster nowadays, or, or what, what, what, uh, whatever. So, um, the SALSA framework kind of. Shows all the ins and outs, like all the, the input outputs where the, you should kind of pay attention. Uh, and, uh, we, we wrote, um, uh, a series of articles and, uh, created attack trees that go in, in, uh, a lot of depth, uh, into, uh, the different sections of salsa. Um, and I, I think. Yeah, so this article is published, it's, uh, kind of the prelude to the, uh, conference stocks, uh, where we have a focus, uh, on sort of the lessons learned of real world attacks. So, uh, I kind of walk through in the, the conference stock by each part of the salsa model and giving a real example of an attack that was in the news in recent years, uh, and learning from, from those mistakes.

[00:21:55] Robert Hurlbut: So you mentioned, uh, attack trees. Could you take us through that, especially against scms, uh, you know, the effort and then also the results.

[00:22:04] François Proulx: Sure. So I think, yeah, SCM is, uh, I, I have a. I made a pretty, uh, exhaustive attack tree. That one was really focused on GitHub, but it, the same research could be applied to other scms. Uh, in fact, it is open source. We used, um, attack tree tool called sidious, which is a pretty cool, uh, little tool that I kinda like because it uses sort of a markdown format, like just text based format, and it creates the, uh, graph v. Like all the nodes balance automatically is just very efficient to, to write the directories. Um, so yeah, I think, let's say, I would say when it comes to source code, the, the, the, when you, you want, where you wanna start is with branch protection. So if, uh, your, the, the source code repo where you store your most critical, uh, uh, source code is not even using branch protection.

[00:22:59] That's really where you wanna start, cuz that's the cornerstone of entering that. The, uh, the default branch, like main branch, master branch that is used to build the, the, the source code that goes into production has been reviewed. So this enforces that There's been peer review and potentially like static code analysis that is performed. Before it gets merged. So the, that, that's really the cornerstone of anything in su supply chain is, uh, branch protection. After that, there's all sorts of tricks and like, weaknesses in the configuration of GitHub, which is highly complex, has all sorts of knobs, uh, uh, everywhere that you can on and off.

[00:23:42] Like one, one of the, I would say, It's a little less, uh, well known thing. There's thing called tag protection. Even so we know sometimes you might like merge to Maine and then later you say, okay, I'm gonna trust like version 1.2, 0.3 with semantic version. So you apply it tag, right? So, but uh, even if you have branch protection, but you don't have tag protection, by default, anyone would rewrite access to the repo can write a tag. And potentially even overwrite an existing tag. So if in your CIO, like, oh, I'm gonna pull the latest 1.2 0.3, it could be overwritten the commit where that, uh, this tag points to could be changed. So that's, you know, something you, you might wanna look at like tag protection. And in fact, uh, one thing I disclosed recently, there was, um, a responsible disclosure that we did as part of a. Research, uh, last, uh, this, this spring on the Terraform modules where, uh, Terraform, uh, if you know, it's the infrastructure as code, um, system, uh, they, they have the concept of modules and those modules are tagged. So when they're in the registry, uh, really all it means is that it has a gi, GI tag and GitHub. And if someone can modify that, that tag, it actually modifies the module. And I was able to demonstrate that there were thousands of modules that were vulnerable to, uh, phone requests. So if I can get up action, you can have kind of, uh, vulnerabilities that you can, uh, through open source. Let's say you have a, a public repo, uh, fork and you can submit open source, um, contributions. Uh, with that, that, uh, vulnerable GitHub action, you can effectively override that tag and compromise the, uh, Terraform module, for instance.

[00:25:38] Chris Romeo: So continuing on the attack tree perspective, I'm gonna kind of go shift in a little bit of a different direction for a minute, just because you mentioned Brooke and being your threat modeling, uh, mentor.

[00:25:49] François Proulx: Mm-hmm.

[00:25:51] Chris Romeo: So attack trees. Versus threat models. What are the compare and contrast for me when, because I, I'm somebody I've, I've not yet used attack trees very much.

[00:26:03] I've looked at them a few times and they look like they'd be very powerful, but I haven't yet grasped how to bring them together. So like, compare and contrast those two things for me. Like are they the same or are they different?

[00:26:15] François Proulx: No, I think it's complimentary. I, I, I think that one, someone with like that has a mature AppSec program should not use only one or the other. It's like really, they compliment each other. Attacks was just from a different perspective, a different angle. Uh, and it kind of maybe helps you get into a little bit more of a structured, very systematic approach of, uh, red versus blue, kind of, uh, like here's an attack, here's the, uh, the, the, the defense. Yeah. In threat modeling, you will also have that perspective, but I think the. Attack tree is really a visual way. Maybe it's a bit more a lightweight instead of a, maybe a threat model that goes into more vrbo. Uh, maybe like a longer document. Yeah, you might have, you know, your DFDS and diagram and kind of look at it from different perspective. Um, but just the attack tree is just a, a way to visualize. Just the sequence of one attack versus another defense. Okay. And then, yeah, but even if there is this defense, there is this other sub attack. So it's really more the, the, uh, going down the, the, the chain of potential, uh, weaknesses. That one small weakness is the crack that leads to the other one and down the path to the ultimate goal. Of, you know, getting what you want, the crown jewel, uh, at the end. So I, I just see that as a complimentary to any other threat modeling, um, activity.

[00:27:51] Chris Romeo: Okay, thanks. Yeah, that was helpful to kinda, as I'm trying to, to move in the direction of attack trees, that's very helpful to get, I'm trying to get different people's perspectives and share those perspectives with the rest of the world. So salsa, let's. I wanna boil this thing down cuz I guess one of the challenges I've had in trying to grasp salsa, I guess today, everything's about me trying to grasp things, um, is like if as a developer, let's, let's, lemme give, let's, lemme give you a scenario.

[00:28:20] Devel, I'm a developer and a small startup. I'm part of a five person engineering team. What do I use salsa for? Is salsa even for me? Is it, does it provide any benefit to me? Or is this something that only works for enterprises? Or like, how does it, how does it change my life as a, as a developer and a small startup?

[00:28:40] François Proulx: I mean, in theory, salsa, what it stands for is, uh, software level. Uh, Supply chain levels for a software artifact. So the idea of the, the word level in it is, uh, really kind of a maturity model, like where you have different stages of maturity. So if you stick to that for a moment, yeah, it's quite more like enterprise grade, like something really mature, like complex setup. Because when you get to level four, It requires to get everything perfect. And it's, I, I think to my knowledge, even those who wrote the model, they even, they have, they struggled to get to Sal Salsa level four in their own, in fact, it was, it's um, originally, uh, pioneer by Google. So they kind of, uh, brought that to the open source security foundation. Uh, and they themselves internally at Google, I mean, I have friends at Google who came to my talk and said, Hey, thank you for this talk. Because internally we have some teams that would, would, uh, would, uh, would get some, uh, some, some, some ideas from, from your talk. So I think, yeah, to, to get to that level of maturity, it's, it's tough. But there are low hanging fruits at level one. Again, like this kind of branch protections really just to enforce that There's code review. If, if there's no code review or no enforcement of it, then you don't know what you're running in production. And then, you know, kind of the software bill of material sbu. So, If you don't know what you're intaking and what, what other dependencies you have in your package, especially those transitive dependencies, you know, downstream, the dependency of the other dependency and. Down the rabbit hole, you don't know what you're running in production. So at the very least have all those tools, which are many, are open source, easy to run, even for small startups, uh, to have that inventory and some basic assurance of what you have. And then if something goes wrong, at the very least you can kind of come back to get commit.

[00:30:49] That you're pretty sure is that one that originated the thing. And maybe you can kind of look at what, what, what went wrong? Um, because, uh, then, then you can look at your CI environment in even like basic ways, just like access control, you know? Very basic information security principle, like lease privilege and things like that. Should everyone on the team have access to modify the configuration of CI environment? Probably not. Um, and just reduce the number of people who have high power access. Say, you know, take GitHub for instance. There is, uh, the concept of, uh, member of an organization, an organization owner. How many organization owners should you have? Probably more than one because the, that guy's on vacation, you don't wanna be stuck, but you probably don't wanna have tens, uh, you know, dozens of organization owners because this is a high power, high power, uh, credential, which can basically modify anything including disabled brand protection. So, yeah, I think it's just your, just normal information security, hygiene, just common sense things at level one. And then later where, where it gets more complex is to really make it temper proof, like temper evident where you, you add the cryptographic signature at every step of the way, and

[00:32:08] then when you consume the, the artifact in production, uh, you are sure of where it originated. And if something goes wrong, the forensics is easier.

[00:32:21] Chris Romeo: You just made me think of something else when you said Cryptographics. So let me bounce this idea off you. I have this idea for, for what I think needs to happen to solve a com, a piece of the software supply chain problem, but I don't know if I'm just, I don't know, grasping at straws. I'm trying to find the right.

[00:32:38] The right English idiom to describe. I, I don't know if, I don't know if this is a good idea or not, or if this would work, but let me, let me give you this idea. So I look at Let's Encrypt and I look at what Let's Encrypt has done for the world of web server certificates. Nobody, everybody has, uh, certificates now, like you see no sites using HTTP anymore.

[00:33:02] And I attribute a lot of that to the fact that Let's Encrypt made it simple. They made it available, they made it free for everybody so that, so that we don't even really worry about that problem anymore. Like of, of all the things, like you said, we've made some progress in AppSec, like this is, there's an example of progress.

[00:33:20] Could we or should we create the same thing for the software supply chain? Why is there not a let's encrypt version for packages where it makes SI signature and provenance and all of those things? For supply chain packages, just as easy as it is for my, my web app certificates that I use today. Like, is that, is that even re am I, is this even a reasonable idea?

[00:33:47] François Proulx: Yeah. And I think that's exactly what, uh, the, uh, the people behind cosign. Uh, if you're familiar with the SIG store project, which is sort of a sister project to. Salsa and open ssf and all those things. Um, one key thing that they brought key to use the word key here, the context of cryptography, uh, um, is, uh, the concept of keyless, uh, signature. So, uh, in most modern build environments like GitHub, action Circle CI or other Travis and whatnot, um, I would say in the past 18 months or so, so fairly recently. Uh, most of those added the concept of open id, uh, connect. Uh, they, they use, you know, open ID connect, which was used like in more like authentication scenarios, but build in to the CI environment so that when the job the, the, the ci, um, build runs, it has its own, uh, authentication.

[00:34:48] So it has like an ephemeral token to prove that it is the virtual machine that ran the build steps. Uh, and can cryptographically prove that it is coming from a virtual machine that is in Microsoft Azure environment. Um, through this Open Id connect token, which is sort of, uh, provided that like, uh, from the sky as, um, in the case of, uh, GitHub action as the GitHub underscore token, uh, um, environment variable or secret. Other environments have something similar. And the beauty of that is that once the virtual machine or the the bill environment can prove an identity, uh, ephemeral identity, then later you can say, Hey, you know, at uh, 3:00 AM on a Sunday. There was a virtual machine in Azure, which builds something, and then you can use it with cosign, for instance, um, to actually sign.

[00:35:48] So it's like the keyless signature. So it's very similar to what she said, like, for let's encrypt, it's like you don't need to manage private keys because, you know, private keys are dangerous. Like you don't want them on your laptop. But the same thing here, you don't want to just have like a private key that you sort of grab from somewhere.

[00:36:07] Like they should have a kms where they keep the private key outside of the boundary of your bill environment and you just obtain a token which is already signed, and then co-sign has a way to use it, um, to, uh, sign the actual artifact using that ephemeral token. So I think that is the. Answer to the equivalent of, of let's encrypt for build systems. Uh, and, and, and there's, there are things like, um, very similar to again, like let's encrypt like the certificate transparency logs. Uh, exactly the same kind of technology is applied with, uh, other projects where you can optionally. Push, um, a transparency log where the hash of some artifact being built in a certain environment can be provably, uh, like put in the public.

[00:37:01] Like you can turn to tho those, uh, transparency log systems and confirm that there really was a hash which was added to that mer Merkle tree. At at 3:00 AM on a Sunday. And it's not just your like, here's a hash. Like, okay, yeah. Where is that from? You know?

[00:37:24] Robert Hurlbut: So a lot of great strategies. Are there some other, uh, things that you could, um, recommend for, uh, developers, uh, and organizations that they could adopt to prevent supply chain attacks?

[00:37:39] François Proulx: There are so many things, like when you start to go down that, that rabbit hole, um, uh, you know, keeps, keeps me up at night. Uh, but, um, I think, you know, When you start by, uh, understanding your dependency tree, I mean, for instance, like one simple thing that I, I would hope that most of your listeners already doing, if they're not stop the podcast, you'll do that, uh, is uh, dependency, like package manager lock files. So like, you know, you may have the package json at the root of your project, you define your independencies, but if you don't have a lock file, Like you may be defining. Okay, open SSL version 2.3 0.3. But that is just like a label. It's just like a, a semantic version. Uh, you need to have a cryptographic hash to ensure that it's really the version of SSL that you expect to be downloading from the CDM, from, you know, NPM and not just like, you know, at any moment, something someone can override, like override the version that is stored in npm. You should be able to assert that. It's the, the hash you expect. So that's, you know, using the, the lock file. Um, and that comes back to what I mentioned before with the Terraform modules. One key thing that Terraform does with providers, if people are familiar with Terraform and providers, it has a lock file. But not for modules. And that's the key takeaway of this, uh, research, is that you don't have the same level of protection to, to guarantee that the module you're running in Terraform is the one you really expect. So there, there are some, um, uh, say open issues in GitHub in the Terraform that's been opened for a number of years to convince HashiCorp to add the same level of protection, uh, as, uh, is provided for, uh, providers to modules.

[00:39:45] Chris Romeo: Very cool. Well, friends, all we're coming to the. End of our conversation, and I want to give you the opportunity to share key takeaway, a call to action. You know, is there something that you want our listeners to do as a result of our conversation or a last thought you wanna leave them with? Let them have it.

[00:40:06] François Proulx: Yeah, there are all sorts of, uh, great initiatives. Uh, if you just look for open ssf, it's a great place to start. Uh, look at pretty much any project they're, they're, they're working on, such as the scorecard project for instance. Uh, it's a way to get just a security posture of your repo. If it's a public repo, you can literally. Click and maybe they've already actually ran. Um, so the security posture on your repo, you can look at the devs dev d dps de V, it's a Google, uh, managed service. They actually scan, you know, open source repos and run the scorecard. On those open source projects and keep the security posture, uh, are archived there. So if you kind of, you know, you're like, ah, I'm about to add this dependency. Some developer on the team is say, how about we add this little JavaScript? Thing you can lo look at dev, dev, dev, look at their reputation. Is it, uh, something that was created like, you know, two weeks ago or it's like five years old with hundreds of contributors? It's still maintained. You know, they're, they have branch protection, all, all those good things, and it's just a very quick sniff test for adding a new dependency, for instance.

[00:41:23] Chris Romeo: Very good. Well, Francois, thank you for sharing this knowledge with us and I know I, I learned a couple of new things and a few things I need to go dig into deeper, like attack trees and I wanna. Look at your example with the SCMS and more, more depth and, and you helped us to understand salsa better. I, that's one I've been struggling with and gave me some pointers for my lessen encrypt for third party or for software packages.

[00:41:48] So I got, I got a lot of homework coming out of this. That's not supposed to happen. I'm supposed to be the one that's giving out the homework, but you gave me a lot of things to think about. So thanks for sharing with us and uh, we look forward to, uh, a future conversation with you about any, about something related to AppSec.

[00:42:02] François Proulx: Thank you.