Secure Code Warrior upskills development teams to help companies stay protected against potential cybersecurity threats.
Connect with Matias on Linkedin.
Shoutout to Lifejacket badge winner Sergey Kalinichenko, who won the badge for their answer to K&R Code for getting an int.
TRANSCRIPT
[Intro Music]
Ryan Donovan: Hello everyone and welcome to the Stack Overflow Podcast, a place to talk all things software and technology. I am your host, Ryan Donovan, and today we are talking about all those security flaws that LLMs can throw into your code. And we're gonna talk about how you could possibly mitigate them with my guest, the Secure Code Warrior co-founder and CTO, Matias Madou. Welcome to the show, Matias.
Matias Madou: Thanks for having me, Ryan.
Ryan Donovan: Of course. So, before we dive into the topic today, we like to get to know our guests. Tell us a short flyover of how you got into software and technology.
Matias Madou: Well, you know, from when I was a kid, I had a real interest in computers. So, I actually went to Ghent University here in Belgium, and ultimately, I pursued a PhD in Application Security, working on static analysis solutions, which is very relevant these days in the world of AI. And with my PhD, I actually moved to the US and I joined, at that time, a very small company who was trying to make a dent with static analysis in finding problems in code, called Fortify.
Ryan Donovan: Hmm.
Matias Madou: So, I've spent seven years at Fortify. I'm working on their static analysis solution, trying to find problems in code. Ultimately, I thought like, 'hey, isn't that super easy? Isn't that super trivial to find problems in code if you never, ever help the developer?'
Ryan Donovan: Mm-hmm.
Matias Madou: And that's essentially how we started Secure Code Warrior with the vision, the mission, with the idea of like, 'hey, how can we help the developer? How can we be there for the developer in writing secure code?' So, that's how I rolled into what I'm doing today and into technology.
Ryan Donovan: So, today, the co-developer isn't always a human being.
Matias Madou: Correct.
Ryan Donovan: And we get a lot more errors in there. What are the sorts of errors that you see over and over again from code gen, AI code gen?
Matias Madou: So, it's true that today, not all code is written by humans. Although, we do see that humans are pairing with AI. It's not really AI that is totally generating the code. I know it exists, don't get me wrong, but the majority of the use cases that we see is where a developer is using an AI system to help him or her with the generation of codes and the creation of applications.
Ryan Donovan: Mm-hmm.
Matias Madou: And in a recent study– so, what are the main things that we still see? It's pretty interesting, I think. The code generation has become to a point where the syntax is correct. It compiles, essentially, right?
Ryan Donovan: Right.
Matias Madou: So, it does compile, and you get some working piece of code. You get a piece of code that can be executed. [If] it's working or not, that's a different question, but hey, it's an executable piece of code.
Ryan Donovan: Mm-hmm.
Matias Madou: Where, a couple years ago, you got something, and it was not always executable. From a security perspective, that piece of research, done by Veracode, is still steady. It's still at the 50% level, so we didn't really progress a whole lot in making or getting more secure code out of these systems. If we dive a level deeper, if you're asking like, 'hey, what do you really see?' Well, within Secure Code Warrior, we did another piece of research trying to figure out what LLM solutions are good or bad in certain types of vulnerabilities.
Ryan Donovan: Mm-hmm.
Matias Madou: And the gist of it– so, there's a lot of detail in that paper, but the gist of it is that certain categories of problems, we think they are gonna go away. So, the good thing is if you say SQL injection, we think we have a really good shot not seeing SQL injection five years from now. I know we said that five years ago as well, and 10 years ago, as well. But right now, our research shows us that LLM solutions, if you say like, 'hey, I need a piece of data from a database, can you help me write a query?'
Ryan Donovan: Mm-hmm.
Matias Madou: Well, nine out of 10, well, even more, it gets you code back, which is using parametrized queries, which is free from SQL injection. So, in the bug category, and really those syntactical mistakes that lead to security issues, we see, really, that LLMs are doing a really good job. Even, we have a shot at not seeing them in the near future. In the flaw category, and more of the design problems, you know, it's kind of hit and miss. There's some progress. There's not some progress. It's sometimes there, but not always. At the same time, there's even a third category of problems right now. The third category of problems are the problems that we didn't have, and that are essentially new. Like, hey, how about hallucinations? Five years ago, nobody knew about hallucinations in AppSec. These days we have to say, well, there's hallucinations. You can get some code back that refers to libraries that do not exist. Anyway, so those are the things that we see the most. If we categorize them in three things, LLMs tend to do better with the syntactical things, and it's gonna be almost resolved. The design problems, from a security perspective, they're still there, and there's a new category of problems that didn't exist. Or the most simplistic one, to just give one example, is hallucinations. That, we need to teach these days.
Ryan Donovan: Yeah. I've heard of people referring to LLMs and code gen–
Matias Madou: Yep.
Ryan Donovan: Almost as super linters, right? So, it makes sense that they catch all the syntax stuff. How do these LLM errors sort of align with the standard, you know, OASPs top 10, the errors that we see from humans over time?
Matias Madou: So, first of all, I like your analogy with super linters. At the same time, one thing that we find pretty annoying is the variability. Meaning, you ask a question today and you get a certain type of answer, and it can be correct. Even so, it can be correct today, and you're gonna ask the exact same question tomorrow, exactly the same, and you get a different answer, which is incorrect. And that is pretty annoying. And because, let's be honest, for software developers, if there's one thing that they get really annoyed by, it is variability. They always wanna do the same things in the same way to get clean code and consistent code.
Ryan Donovan: Yeah. I mean that's the nature of the LLM, right? It's a non-deterministic system.
Matias Madou: Yep.
Matias Madou: But I agree 100%, but there's a lot of people that do not get that particular point. They do not get that point. So, linters, they will always—it's rule based—they'll always tell you the exact same thing because it's based on rules, and it's not gonna change. If you ask a question today and you have the same linter 10 years from now, guess what? It's gonna give you the exact same outcome. That's a big difference. And people do not understand that big difference today.
Ryan Donovan: Mm-hmm. You mentioned using the LLM as a pair programming partner.
Matias Madou: Yes.
Ryan Donovan: You know, the LLM puts out, you know, certain errors. This puts a lot more onus on the developer to really be able to catch the errors, the flaws in this pair programming process, doesn't it?
Matias Madou: It does. And there's something weird going on. I think we collectively get dumber because we lose critical thinking, and it's– by the way, it's not throwing a stone at developers, in general. The population is getting dumber because we just believe the LLM, we just believe, as you just said, a non-deterministic system that is just putting the next best word in front of you, and we believe it. So, the things that we see is that, increasingly, developers that even know their stuff are believing more the LLM than trusting their own gut feel[ing], and their own, essentially, knowledge. Not only gut feel, but their own knowledge, their own skillset. They're doubting their own skillset because they trust what an LLM is saying. That is pretty frightening. I think that's very frightening, that you trust more a system and non-deterministic system that predicts the next word than your own skillset, and that is something that, you know, we need to relearn essentially. We need to relearn to trust ourself. And the more skilled we are, the better we can use those LLM systems. Essentially, the interesting analogy that I like to use, or the test that I like to use, is, let's say you know about a particular field. Ask ChatGPT a question about that field, and you're gonna be like, 'ooh, that's not entirely correct. And that's a little bit, you know, over here and over there.' But then do the same thing about a field that you do not know. Well, guess what? You quite often just believe the system, right? You've gotta be like, 'okay–'
Ryan Donovan: Sounds right enough, right?
Matias Madou: Yeah. But so, that is a question. When are you asking a question to those type of systems? When you know your stuff, or when you do not know your stuff? Quite often, it's when you do not know your stuff and you're just believing the system. So, it's plain out dangerous. So, our philosophy is really like, hey, those systems are really good systems for people with knowledge, people with skill, and they have to be really skilled because they need to trust their own skillset more so than trusting the LLM system. So, our philosophy is LLMs, AI systems, all of that stuff, it's not gonna replace humans anytime soon, especially not in the coding world, because it requires a skilled person to essentially make those applications. It can be a tremendous help, but it's not gonna replace them.
Ryan Donovan: Yeah. I think we've seen that play out in some of the research where they've said that the people getting the most value out of these AI code gen tools and vibe coding are the senior developers, the people who already know a ton about best practices and all that.
Matias Madou: Yes. So, if I may pick up on that one. So, that is true, but that does not mean we're asking less of those senior developers. We say, 'hey, well, you've written three features, can write 10 features?' We're not gonna say, 'oh, we have two senior developers. We only need one because the one can do the work of the two.' In organizations, quite often they're like, 'well, guess what? There's these bazillion feature requests.'
Ryan Donovan: Sure.
Matias Madou: We only expect more output. So, by the way, my gut feeling is that, right now with AI and with AI pair developer, we called up with the demand. It's not that, you know. We have an oversupply of developers right now. No no. We just caught up with demand because right now we, had so many requests the developers were just snowed under. These days, I think with AI, we have a shot at finally meeting that demand and being able to code all the requests that an organization is asking for.
Ryan Donovan: Right. Yeah. Everybody has their big wishlist.
Matias Madou: Oh yeah.
Ryan Donovan: Yeah. But we talk about the volume of code that AI is putting out, right? And that puts a lot more on the code review cycle.
Matias Madou: Yes.
Ryan Donovan: And I've heard people talking about using AI tools to help with that code review to spot security flaws. Do you think that is a workable solution?
Matias Madou: Ooh, that's an interesting one because what we see quite often is also the inverse, where we feel that a developer is becoming more and more the reviewer.
Ryan Donovan: Mm-hmm.
Matias Madou: Not the AI solution becoming the reviewer of developer code, but the other way around. If you ask an AI solution or an LLM like, 'hey, I wanna do something,' you essentially become the reviewer of the AI code. I think it can work both ways. It can work both ways, where me as a developer, I write some code and hey, you know, I gotta check that with an AI solution to figure out if I could do better. The other way around, I'm not sure how to program this. I ask an AI solution to get me a piece of code, and then I become a reviewer. The one thing that I've learned personally, because I have a couple of side projects, you know, and home automation stuff, is you have to damn know well your code to go faster. Because quite often, you're reviewing in code from something else. It's not somebody else, but from something else.
Ryan Donovan: Mm-hmm.
Matias Madou: You ask like, 'hey, I wanna do this.' You get some piece of code and then you're like, 'ooh, how does that work over here?' And by the time you're, you figured it out, you could have written it yourself.
Ryan Donovan: Right.
Matias Madou: So, it's give and take. It's give and take.
Ryan Donovan: Yeah. And editing something, and understanding it, and making it work better is often as hard as writing it the first time.
Matias Madou: Yes, and I think that's something we genuinely underestimate. People say, well, with certain type of companies, you can essentially create full apps. and that is correct. But then, you start at– that's the critical point. If you wanna add new features, if something is broken, if you wanna fix something, then you suddenly need to review somebody else's code. And you need to dig deep into, well, 'what's going on here? And how should we modify that?' And from that moment on, it's becoming increasingly harder to use AI solutions to modify the code. So I think that initial step goes really, really fast. The whole vibe coding, I can see why it's working, because you're starting with a clean sheet. But once you have an app, once you have an entire system that interacts with stuff, it's becoming increasingly harder, to do that rapid development with AI. I'm not saying it's impossible; it's increasingly harder.
Ryan Donovan: Right. Yeah. There still the human there. We mentioned at the beginning the sort of easy syntax security flaws are the ones that the AI gets. It's the harder ones that require a little more thought process, that are the ones that are getting through. How can developers train themselves to be able to spot these harder systems problems?
Matias Madou: I think it all starts by having good knowledge about the language and framework that they're coding in. Maybe the best analogy that I can use is when I started coding, I made simple mistakes like, 'hey, every Java line has to have a semicolon at the end.' Then you make syntax mistakes like that. Once you get through that hurdle, you learn about, 'hey, how can I make functions? Oh, different files, architecture.' And it's only at the end of the journey that you start to think about, like, there's even hidden functionality in my application with security implications. Like, huh, people can actually empty my database. That was not the functionality that I wanted. So, it all starts by knowing the language and the framework really, really well. Up until the point that you essentially have that knowledge about how to create secure applications. So, to me, that is key. To me, it's key to have a very good understanding of the language and framework, including how to securely code in that language and framework.
Ryan Donovan: Yeah. So, I know a lot of folks—in the last 10 years was the rise of the bootcamps—and a lot of folks sort of learning to program from that. Is this gonna change how bootcamps work?
Matias Madou: Ooh. No, I don't think so.
Ryan Donovan: No?
Matias Madou: 'Cause let, let's be honest. Even at universities, if you look through the curriculum, there's not a lot– there's security, don't get me wrong. You know, you learn about crypto and SSL, that type of stuff. There's not a whole lot about the funky stuff, like cross-site scripting and SQL injection. So, I don't think it's any different. It seems to be a skill that needs to be thought after learning, after university. It's only once you enter a workforce and you enter a financial institution or a tech company that they start to care about the secure functionality of the pieces of the code. So, learning how to code, I'm not sure if that's going to change. It's all about once you enter the workforce, and let's be honest, certain organizations truly care about their reputation, about writing secure code. Other organizations are just winging it until something breaks and something goes wrong, and then suddenly, they do care. So, it all depends when you get in touch with that.
Ryan Donovan: Yeah. I mean, I think there's a little difference between the, you know, the bootcamp and the university education. University education, you get more of the theoretical computer science, right? You get the math, and then you get some other courses to sort of round you out as a thinker.
Matias Madou: Yep.
Ryan Donovan: Do you think there's gonna be bootcamps for the sort of critical thinking, systems thinking, that this can be more prevalent as the AI coding sort of becomes more prominent?
Matias Madou: I'm sure somebody will jump on that bandwagon. I'm sure they will. I'm not sure how that will shape out. I'm trying to get my head around– you know, there's definitely an area where they can do and use all of the AI systems, but how that will pan out with security? I don't know. Because the paradigm still stays, right? You still need to know about the basics. You still need to know about building secure applications. And then, AI is just a layer on top of that. How can I use AI to make that faster, better? That type of stuff.
Ryan Donovan: Yeah. And you talked about– you learn about a lot of these security flaws, the cross-site script scripting in a job, right? It's sort of an apprenticeship program. One of the things I've heard is that, you know, junior developers are gonna be really hit hard by the AI code gen. How can we sort of improve and replace these apprenticeship programs that are the sort of junior developer role?
Matias Madou: It's a very interesting question that one, because I hear different things about juniors and seniors. So, the initial research that you also refer to tells us that senior developers can take most advantage of AI, and they essentially go faster, which is just normal. They know about the stuff. They know how to poke an AI solution, and so on and so forth. The latest story that I've heard is where they did some sort of a competition between senior people and junior people on the usage of AI and the creation of secure applications. And in that scenario, the junior people were actually doing better than the senior people because they essentially embraced the AI movement, and they're trying to figure everything out, and they learn, essentially, faster. Where the senior people are more defensive, and they say, 'well, I'm not gonna use this. You know, and , I know what I do, so I do not need that.' So, I don't know where this is going, but in certain pieces of research, seniors seem to be at the advantage, and in other pieces of research, juniors that really embrace it and want to go fast, they seem to be at an advantage where the seniors are essentially stuck in their own thinking.
Ryan Donovan: Right.
Matias Madou: Where, the philosophy here is, you know what, well, let's say you, we all know a marathon. It's running, but let's say the rules change tomorrow, and tomorrow you can use a bike to do a marathon. Well, if you're the best marathon runner in the world, you can fight this and say, 'I'm not gonna do that.' You know, 'I'm stuck in my old paradigm. I'm not gonna do that.' You know? Marathon is running.
Ryan Donovan: Right.
Matias Madou: It doesn't matter. Like, if there's a tournament and people are using a bike, he is never gonna win. So, either he or she has to change and take a bike, or it's a lost game. And I think that applies a little bit to that newer generation, older generation, type of philosophy that I was just referring to.
Ryan Donovan: Yeah. That almost, like, beginner's mind advantage, right?
Matias Madou: Yes. They do not know anything better, so they go full in, and 'oh my God, it works!'
Ryan Donovan: Yeah. And I've heard that the skill that's gonna be most important to programmers is learning how to learn. And I think, you know, programmers have always been learning all the time on the job, but at some point, you build up a sort of hard crusty exterior of best practices. Like you said, here's how the game is played. Do you have any ideas on how to regain that beginner's mind for the seniors?
Matias Madou: It's hard. I'll be honest, it's hard. We see that quite often as well when we try to upskill people in an organization. There are people that are receptive to this and have an open mind, and quite often these are people new to an organization, younger people, quite often, not to stereotype, but that's quite often they are open-minded and they go in and they wanna learn. It's really, really hard. 10 years into a company, they're like, 'I know what I'm doing. I do not wanna do this. I know what I'm doing.' Well, quite frankly, it's not always the case. Some of them do, don't get me wrong. Some of them do, but some of them, they never did it correct. And it's really, really hard to change, you know? That's the thing. You cannot learn a pony new tricks, or– uh, sorry, I don't know.
Ryan Donovan: You can't teach an old dog new tricks.
Matias Madou: An old dog new tricks. I'm sorry.
Ryan Donovan: No worries.
Matias Madou: Lost in translation. Lost in translation.
Ryan Donovan: Yeah, yeah. So, when you're thinking about helping developers understand how to approach the AI and the security defense, how do you– what's your process there?
Matias Madou: Our philosophy is that we should help a developer, making sure they know what's out there and how to use these solutions. Essentially, making sure they have an environment where they can thrive using AI. And that means a lot of different things. First of all, it knows what the good things are about AI, but also, what the unknown or bad things are, like IP, for example. Like, 'hey, how does it work with IP?' So, the good things and the bad things. That's where we start, from a higher level. Hey, Mr. or Mrs. Developer, this is what you can expect, and this is the advantages and the disadvantages. Second thing is, we wanna bring back critical thinking to an engineer. And the way we do that is actually very simple. So, we are gonna pretend they've asked a question to an AI solution, and you get an answer. And then we are gonna ask the question, 'do you accept that? Do you check that in?' Because by the way, this is your code, right? This is, you cannot blame the AI. This is going to be your code. Your name at checkin is there. It's not gonna say GPT-3,4,5, whatever. No, no, no. It's gonna be Matias Madou in there, and you're gonna be responsible. So, do you accept that or do you know better? And, you know, it's some sort of a role playing where we go through it. If they say, 'well, this is correct,' we're gonna say, 'are you sure? Have you heard about SQL injection? Maybe there's something there.' Or if they know and say, 'no, no, this is wrong.' We can also go through it, like, 'what do you think is wrong? Is it only SQL injection? Is it also hardcore passwords, whatever?' Like, it's going through that scenario. Next step, we do wanna make sure they know about the latest and greatest MCP servers, that type of stuff. Again, the advantages, the disadvantages. So, ultimately, we hope that they have an environment where they can thrive. They are skilled enough to use those tools to their advantage, and they go faster. We wanna see an ROI, we wanna see them go faster, and we also wanna measure that. So, that is essentially our philosophy.
Ryan Donovan: Yeah, looking at the code and sort of really thinking through it is pretty key, because if you write the code yourself, you think through the code pretty intensely. But if you're just using the code gen, you just get the product.
Matias Madou: Correct.
Ryan Donovan: It's so easy just to ship the
Ryan Donovan: product instead of think[ing] through it afterwards.
Matias Madou: Yep. And that's problematic, right? We've seen examples, if they're real or not, on Twitter, for example, where people say, 'hey, you know what? I started this business very, very fast,' and then it goes under really, really fast as well. So, yes.
Ryan Donovan: Yeah. I think we've all seen that Twitter thread.
Matias Madou: Yep.
Ryan Donovan: Chortle to ourselves. So with the future of vibe coding, with the future of, you know, entire products being written within an AI agent, besides knowing everything about the security clause, is there something they can do to help train the agent? Some sort of context?
Matias Madou: I thought you were gonna go in a different direction here, which is, what we also see is that suddenly we have a lot more developers because what you've mentioned is like, hey, anyone—marketing, sales, finance—they can essentially start building apps. They are creating code. For example, financial institutions.
Ryan Donovan: Mm-hmm.
Matias Madou: They were brick and mortar, and then over time, they morphed into more of a software company because they realized like, 'oh my God, half of my people are software engineers. I did not know.'
Ryan Donovan: Right.
Matias Madou: Well, it's even worse right now because right now, now it's like, 'oh my God, even our finance people are writing code. So, they make it even worse. Right now, they're really a software house because 90% is using AI to generate code and create and spin up apps. So, the nightmare essentially continues. I forgot the original question, but that's where my mind went.
Ryan Donovan: Yeah, yeah. Well, that's either, you know, there, you can either get your finance folks to memorize and understand all of the security flaws, or you can get some additional context into the agent itself.
Matias Madou: Yes. Oh, yes. Correct. And I think there's definitely a market there. I'm trying to get my head around how much will LLMs improve over time, and will that still be necessary two, three years from now? So, I hear you. So, I think today, LLMs are not at the point where they are perfect. Can you make them perfect? You know, it's non-deterministic, so no.
Ryan Donovan: Right.
Matias Madou: That's the short answer. You can have a long debate, but ultimately, the answer is still no. You can make them better. You can make them better by crafting some rules and make them better. The question is, how much will they still improve, and are these additional rules material so that there's essentially a trade-off? An interesting other point here is, yes, you can train them, you can use better models, but I'm not sure if people know about this whole model of AI, but this is heavily funded by VC money. What we're using today is not what we're paying for. If you pay 20 or 100 dollars for– I just bought Cursor, for example. If you pay that amount of money, that is not the amount of money they are spending to–
Ryan Donovan: Right.
Matias Madou: They're burning money and, essentially, they use a model where they burn money, and they use chips where they can take a loss because they wanna make sure they sell as many chips as possible to get some more chips, and yada, yada, yada. So ultimately, when we're spending a $100, well that can potentially cost $5,000, and I'm not exaggerating. Throughout the chain of all the people that touch it, and all wanna make sure you're using their stuff because they're hoping to sell more of it later on. So, that's gonna be a good question. Later down the track, will that cost go down, and essentially, will we be able to pay $100, and it's costing $100? Or will that imbalance proceed? And then you will have, essentially, the question will be asked, 'will you use model 5 or model 3.5? Because there's a difference in pricing over here. And in that model, then you can potentially start using these type of rules. So, it's an interesting debate because there's a lot of money involved over here. A lot is funded. What we're paying today is not really the actual cost. They're all betting that it's gonna become cheaper and these models are going to improve, but it's all yet to be seen how fast this is going. Because ultimately, we all think this started in 2021, but that is not the case, right? Accelerated in 2021. But even—now, I'm gonna date myself—but even when I was at university in 2003, well, there was a lot of people doing AI and ML stuff. So, it's nothing new.
Ryan Donovan: I took a AI programming course in 1997 and had neural networks in that.
Matias Madou: You're dating yourself as well, my friend.
Ryan Donovan: I'm dating myself. Oh, don't be afraid. We get a lot of gray beards on this show.
Matias Madou: Oh yeah. That is true. That is true. I think what will be important in the near future is trying to figure out who made what. Like, who was made by what or who, because we do not know. What's made in code by an LLM? What type of LLM,? What is made by humans? What is the risk involved in that piece of code? What is the confidence that this is good code? So, I think that's gonna become increasingly more important in the near future. So the way we develop software in the very near future is changing. It's changing every six months. But it's gonna be very, very important to figure out all the pieces of the puzzle on how software is created and who or what created the software. I think that's gonna be very important.
Ryan Donovan: So, the most important command of the future is gonna be git blame ?
Matias Madou: And then they're all gonna deflect on the LLM. Oh, yeah–
Ryan Donovan: Yeah, yeah.
Matias Madou: It was that thing.
Ryan Donovan: It's that time of the show, ladies and gentlemen, and distinguished guests. Where we shout out somebody who came on the Stack Overflow, dropped some knowledge, shared some curiosity, and earned themselves a badge. Today, we're shouting out a life jacket badge winner. The baby lifeboat, somebody who came to a question that had a negative two score or less, and dropped an answer that brought up both the question and earned themselves five points. So, congrats to Sergey Kalinichenko for answering, 'K&R Code for getting an int.' If you're curious about that, we'll have it in the show notes. I'm Ryan Donovan. I host the podcast, edit the blog here at Stack Overflow. If you have questions, concerns, topics, et cetera, please email me at podcast@stackoverflow.com. And if you wanna reach out to me directly, you can find me on LinkedIn.
Matias Madou: Hey, my name is Matias Madou. I'm the CTO and co-founder of Secure Code Warrior. You can find me on LinkedIn, Matias Madou, and you can definitely find our company Secure Code warrior at www.securecodewarrior.com.
Ryan Donovan: All right, thanks for listening, everyone, and we'll talk to you next time.