On today’s episode, Ben and Ryan chat with Laly Bar-Ilan, Chief Scientist at Bit. They talk about representing code in LLMs at the component level, keeping a code base clean through smart code reuse, and whether vibe coding will eventually succeed as a way to build production-ready software.
Episode notes:
Bit lets you generate composable software simplicity, speed and quality.
Connect with Laly on LinkedIn.
Congrats to Lifeboat badge winner and blog contributor Charlie Martin for their answer to How does this proof, that the halting problem is undecidable, work?.
TRANSCRIPT
[intro music plays]
Ryan Donovan Hello everyone, and welcome to the Stack Overflow Podcast, a place to talk all things software and technology. I am Ryan Donovan, and I'm here with my on-again off-again co-host Ben Popper. How you doing, Ben?
Ben Popper Hello, hello. I'm on again. I'm doing great. You just got back from a big conference, right?
RD I did, yeah. Talked a lot about AI. Today we're going to keep the train rolling. We're going to talk about AI and code quality and composable software. We're using code with AI.
BP Can't talk about technology these days without talking about AI. So we're just talking about technology, just talking about software.
RD So we have a great guest today, Laly Bar-Ilan, who is the Chief Scientist at Bit. Welcome to the program, Laly.
Laly Bar-Ilan Thank you, Ryan, and thanks, Ben, and thanks so much for having me here.
RD Of course. Top of the show, we like to get to know our guests. How did you get into software and technology?
LB Okay, so it was an unconventional path.
RD If I had a dime for every time a guest said that.
LB I'm relieved to hear that. I felt a bit of an infiltrator for several years, but after 20 years almost of coding, I don't feel that way anymore. But actually started with a master's in linguistics and cognitive science, and then I learned how to code. I actually taught C++ after learning that and then when the whole computational linguistics took off in the early 2000’s, I joined the trend and became a software developer mainly with NLP. So I did many things like text evaluation, translation, grammar checking, then I got into code semantics, code generation in the past couple of years. So that's my unconventional past, like everyone else's.
BP And tell us a little bit about the company that you work for. I was doing some research and it seems a lot of the company's history is in the world of composable software and architecture.
LB That's true. The company started 10 years ago, I joined nine years ago. We've been doing composable software ever since it was founded. We strongly believe in composable architecture and so we've always built in a very modular and reusable way. We've always built with components, both back end and front end. I mean, most companies today probably use components on the front end– React components like buttons and headers and all of that, but not a lot of people associate components with back end. And we do everything. All the code base is represented as components, so every microservice, every entity, user or item ticket is represented as a component, every DB handler, anything. Basically the principle is everything that we think other people can use in other contexts becomes a component. We've been doing components long before we introduced AI into the mix.
RD It sounds like you're taking encapsulation to the next level here.
LB Each component encapsulates its own implementation. So basically all we've been doing all these years is just composing components together. That's what we aim for AI to do. Instead of generating code snippets every time from scratch, it's more about taking components, existing ones, and composing them together.
BP I saw that on your LinkedIn you had given a talk. One of the things that I thought was most interesting you had up top as one of the bullet points are, “Components are better than tokens.” So this idea that the AI might generate something, but we want it to generate something that's reusable, that's fixed and understandable, a little bit more deterministic. What do you mean when you say components are better than tokens?
LB Okay, so let's start with what are components exactly for us and what are tokens– just start with the definition. Components are completely independent pieces of software like mini repos. They have their own implementation, their own API, their own docs and tests and version control that allows them to be runnable and importable and be composed with each other because they each have their own APIs. As for tokens, tokens in code generation are actually reserved words like if-else and for, and of course, variable names, class names, et cetera. Okay, so these are tokens. Basically, AI today knows how to analyze sequences of tokens and then generate sequences of tokens each time predicting the next most probable token. This is how it generates code. So this is the difference between components and tokens. Why are components better? Think about it. We want the AI to be able to think like us in the same categories that we do. I think it's like looking at a topographical map versus a political map. When AI sees the code, when we see code, we just see lines of code and the AI sees lines of code, but we interpret them completely differently. What it sees is sequences of tokens, whereas what we see is features, product, and the whole logic. We see the components and the connections between them. This is something that's not explicitly accessible to the AI and we are making it accessible.
RD That begs the question– how are you making it accessible? Are you tokenizing at the component level?
LB We are representing the code at the component level. Our representation of the code, both for us and for our customers is to actually take the entire code base and represent it component graphs. We don't only represent the components but also the connections between them, the dependency connections, so we have a whole graph. What it actually represents is a map of the entire business and product functionality in the organization. It's a really clear, visible map. You don't have everything just hidden in separate repositories, so it's a huge advantage.
BP I was just going to say some of the other bullet points that I thought were interesting, which maybe gets to what you're saying was that holistic is better than local, access to the entire code base not just a local repo, accurate context. The key– reuse is better than generating each time, which gets us back to components. So where it seems it's moving with code generation, it's like RAG in the sense that the first thing we really want to do is make sure you come in and understand our code base, our idioms. Some code is idiomatic because of the way it was built or the language you chose, decisions that were made that you may regret, but you don't have the time to pay down the tech debt now. If you can get the AI to understand that these are things we've already built, just reuse these instead of writing them again from scratch, the end product is going to be so much smoother of an integration.
LB Exactly, and it's not inflating your code base. Because we represent the components on this layer of RAG, we don't represent the token level, we don't even represent the implementation. All we have indexed is actually the APIs and some metadata like docs. So the AI doesn't even have access to the implementation. All it uses is the APIs of the component. Think about it. In order to put together Lego bricks, you don't need to know how they look inside. All you need to do is understand the little studs and put them together. This is exactly the same principle that we use. In this way, you talked about accurate context which is better than big context. We know that a lot of code generators today have really big context windows. I heard of an LLM that actually has a hundred million tokens in a context window, and I'm thinking this sounds impressive, but on the other hand, it's a lot of noise. Most of this information you don't need. Let me give you an example. Let's say you ask the AI to build you a header component. The first thing it does is use the RAG in order to find whether there's already a header component somewhere in the code base. If there is, then it uses it and end of story. If there isn't, what it does is build a virtual dependency graph. It says, “Okay, what does a header component usually use?” So what do we usually have in headers? We have a logo, a menu, a search box. It will generate this graph, then it will go search for these components. It's a recursive process. Whatever it can find it uses, and whatever it doesn't find, it generates. One of the most important things we do for companies today is prevent their code base from inflating. Think about it. You know how there are duplications today? You have one team working on something and another team working on something, and they both develop the same functionality and they don't know about each other, but now we have this with AI which does it so quickly and inflates the code base quickly.
RD I wonder, you talked about reusing the components. Is there a risk of violating the sort of DRY principles– the don't repeat yourself– where you're copying the same bits of code in the same places, or is there a specific connector piece that it uses?
LB So it's interesting that you say that because the DRY principle is our main principle. Reusability, don't repeat yourself. AI today is inherently WET. Do you know what WET stands for?
RD It's the opposite of dry.
LB So it's ‘Write Everything Twice.’ This is what AI does today. This developer tells it, “Okay, generate a button,” and it gladly does so, and then another developer in another team asks for it, and then there's another button in the code base. The idea is to see the code base holistically like a live organism, and then you extend it intelligently. You only extend it if it needs extension in the functionality. You only add what's not already in there.
BP Yeah, exactly. Paul Ford, who used to be one of my co-hosts on this podcast and who writes well on software development said recently that the thing that you don't see most people using LLMs for when it comes to code these days is focusing on scope, and how do you prevent scope creep, not just to ship but to simplify and prioritize and focus on why you exist in the first place. And so I think as we see code able to generate services like Cursor able to generate massive volumes of code in minutes, what you really need to focus on is what do I need, what can I reuse, what's going to be performant and give the end user the product experience that they want?
LB True. Exactly, and it makes the code base more maintainable in the long run. And you know how we're always talking these days about keeping humans in the loop with AI taking over more and more development tasks? Think about how a code base would look like if we just let AI run rampant and generate whatever it wants from scratch every single time. How much control will we have as humans on a code base going forward?
BP Ryan, I want to let you jump in, but just to finish that thought, the head of Anthropic, Dario, was at a conference recently. He said in six months, 90% of the code produced will be written by AI. Now that doesn't mean we'll have less developers working on these things and figuring out how to use that code and how to ensure that the code that's generated is the best it can be, reusable, whatever that may be, but 90% of code will be generated by AI. Just curious, Laly, what do you think of that prediction?
LB Okay. I have to agree. From what I'm seeing and actually developing such a tool myself, I actually agree with that. How much of it will be good and maintainable in the long run? That's a different story. What would software development jobs look like in the future? I think that's a fascinating question. I'm trying to figure out what I'm thinking about that. What do you guys think about that?
RD It seems like a lot of people already are generating a lot of code. I was going to ask about the whole vibe coder movement where people are churning out a ton of code and it's not going through the same checks, they're not really understanding it. Stack Overflow was always a place to copy and paste code, but now it just seems people are just pasting it. They're not even looking at it to see what it is.
BP I mean Laly, I'm very interested in it. I do think one thing it's unlocked obviously is the ability of relatively non-coders to start to create apps and services, some of which do find product market fit and get some traction, so that's interesting that more people are able to be software developers. I do think it will accelerate the performance of junior engineers, but companies will require less of them, and senior engineers will really be able to take advantage of stuff, and that, to your point, they'll act more like product managers. Here's what I'm looking for, here's why I think we should build it this way, generate the code and the tests, and they'll come back and do the pruning and the editing, and figuring out, of all the code that was generated, what here is the most useful, impactful.
LB And orchestrate and evaluate. I mean, Ryan, you referred to vibe coding. There was this really smart guy on X who said that everybody's doing vibe coding, but let's see them when it comes to vibe debugging and vibe deploying. That's where it gets tricky.
RD There was another person that was like, “I just built my app in Cursor,” and then everybody jumped on it and he's like, “Everybody's breaking my app.” There's no security to it.
LB Exactly.
BP No, I mean there are these funny examples now of people, and it's funny sometimes I feel bad for them. “I just spent four months in Cursor. I built this incredible app. I made a change and everything got messed up. Is there some way to get it to go back to the way it was?” And this person has never heard of version control. They just sat down with Bolt and Lovable and Cursor and started going because it was fun and they were getting places. They didn't know some of the basics that'll save you when things go off the rails.
LB Exactly. So it's good for proof of concept sort of things. I don't think it's enough, definitely not enough for enterprise grade code which is actually maintainable and secure in the long run.
[music plays]
RD SREs, slow MTTR impacting your team's productivity? Mezmo’s free trial services key patterns and anomalies fast with AI-driven profiling, helping you resolve incidents quicker. Improve your response time. Try Mezmo free at mezmo.com/freetrial.
[music plays]
BP I wanted to ask– there was a blog post that your company put up about the launch of its new AI tool, Harmony, and in that, the company sort of tells the story of hitting a wall, coming to a point where maintaining 30+ microservices was actually creating complexity that limited your scaling potential, that there was a sense that this was a collection of disconnected fragments. Can you talk a little bit about hitting that wall where composability started to feel like as much of a burden as an advantage? Many people have had the same experience with the monolith. The monolith has its pros and cons. Composability has pros and cons. Talk a little bit about some of this stuff that's in here– long feedback loops, complex deployments, and then what the idea is behind Harmony to solve that.
LB So there's Harmony, which is a part of the implementation, and then there's Hope AI, which is our actual AI code generation tool.
BP I was reading on the Bit blog this sort of story you had told recently. This blog is from two months ago, and I guess it's from the– is Ron the CEO?
LB Yes.
BP And so it was sort of talking about this idea that the company had been built around composability and had a lot of success there, but now was feeling some of the pain points of that approach, that certain bugs proved intractable, complexity that limited scaling, and that Harmony was next step at your organization to move composability forward and to connect it with things like Hope AI, so that you could get around some of the friction points that you had found. Does that make sense?
LB It does make sense. I have to say that I'm a bit less involved in the Harmony team and more involved with the Hope AI team. It'll be easier for me to answer questions about that.
RD You said that you started in computational linguistics. How is programming languages, how are they like a spoken language?
LB So they have their own vocabulary, like natural languages. They have their own grammar in the sense of a set of constraints of what words can go together with other words to form a bigger meaning, but they are more constrained. They have limited, more limited rules, more limited vocabulary, and of course are more deterministic and less vague. The main difference between natural language and code language is that natural language is many times vague.
RD There’s still a little bit of undefinedness in some of the compilers, but they sort that out with every version.
LB That's true. Little islands of that. It was still invented by humans that are undefined themselves in many ways.
BP So I guess looking forward, what are you excited about working on and building this year? And do you think, to what we were talking about earlier with 90% of code, do you think the way that you and your teams are going to be working this year will feel significantly different than it did one or two years ago? So just to rephrase the question, what are you excited about working on and do you think within an engineering organization like your own, the way it's done is going to be significantly different this year?
LB It's already changed dramatically. It started with the introduction of code generation tools in general, and then when we developed Hope AI, now we're using it all the time and it's pretty crazy to see. I mean, you can give it some simple prompt and it can generate a whole platform, and not only that, because we have the component graph that I talked about earlier, it can actually validate its output. So it generates the components, it reuses whatever existing components it can find, and then it embeds that in the existing component graph of the organization, and then it does something which I think is really remarkable, which is run simulations of the CI process in order to see how it impacts the dependence. So if, for example, it had to make changes in existing components, it can calculate the subgraph of all their dependents on the graph and then run the build recursively for each of them and see what happens. So basically before you push, definitely before you deploy anything, you can see how it's going to impact the rest of the code base, which is pretty awesome, I think.
BP It worked on my machine. Hey, it worked on my machine.
LB Exactly.
RD You just have to push it and let it fail in production or in testing. It almost seems like we're moving to a future where there are multiple staggered specialty LLMs– one that writes your unit tests, one that does the refactoring, one that creates a composable software. Do you think there will be a point where somebody is just whispering into a microphone and actually having it be successful? Will vibe coding succeed at some point?
LB I'm sure it will. I have to say, I'm sure it will. I mean, think about it. A lot of these things, a lot of the things that we do, even checking for security and evaluating output and all of that, eventually it is repetitive. It's not endless types of tasks, and whatever is repetitive can be learned by AI.
BP So Laly, I have a thesis and you can tell me what you think. I do think, to Ryan's point, what's happening with model context protocol or if you go into Lovable today and you see the integration with Supabase is, I can vibe code just saying what I want and it says, “Do you want me to figure out the backend for you? Do you want me to start hooking things up and connecting APIs and creating buckets?” And so eventually I would assume that service would then say, “All right, it's easy to integrate with a database. Would you also like to integrate with a security provider? Would you also like to integrate with Shopify for your e-commerce side?” They'll take you to a service that has done this, like you said, at an enterprise-quality level of software, and your vibe code front end and your desire to build a product that does X, Y, and Z can be immediately hooked into these services that are at an extremely high level of reliability. I think that takes care of one piece of it, but the two things that we don't see luckily LLMs doing yet, and which I would like to get your take on, is net new creating some insight like a new algorithm that's going to drive forward the kind of code we create, or the next transformer, or whatever that may be. They're not creating net new things. They're responding to our request to create features. And they don't have taste, and taste and product are important. The way the product is presented, the way it's marketed, just the way that you respond to user feedback can be huge. But I would like to get your take on what you see them capable of and if you think, in a year, just like where we were one year ago is now where we are today, there'll be an LLM for taste or for innovation.
LB I don't know if to call it pessimistic or optimistic, but it depends on which side you're on, but I do think that we can quite easily mimic taste for AI in the sense that it can learn user behaviors and responses and do A/B testing at scale that no human can do. And in that sense, it can definitely learn what we like and what we don't like. So maybe it won't have taste like we define it, but it will be able to know this interface is better than this one. It looks better and people will relate to it better.
BP Unfortunately, I think you're right about that.
LB And as for algorithms, I agree that it's more complicated, but one of the first things I tried with ChatGPT I think was, “Hey, can you come up with new data structures?” and it came up with new data structures. I don't really remember what they were, but I was like, “Oh wow, this is really surprising.” I do think that, especially with transfer learning where you can actually take one domain and learn for another domain. Think about it. If they can come up with solutions that work in engineering or in medicine or in law, I don't know, and then do the abstractions and then transfer it to another domain, why not?
RD It's almost like it isn't really a new solution, it's just applying an old solution from another domain.
LB That's one way that we are creative as humans, our combining solutions. This is how we invent things.
RD Well, we are coming up on 30 minutes here. Is there anything you wanted to cover that we didn't get a chance to cover?
LB Maybe just one more thing I was thinking about recently. And look, with AI writing 90% of our code like you mentioned before, Ben, it's generating so much code from scratch and it's so redundant. What if open source could be componentized in a way that we'd have this global component graph and each component can be improved over time and can be reused. The business functionality and the product functionality just repeats itself. We're not really inventing a lot of new things in that sense.
BP No, I like that idea. And a lot of companies now are considering the costs of how many tokens are my engineers requesting every day, and they start to add up. And so if it was instead to ask it to go to the component library and pull in things that are rated best in class that have the most GitHub stars that we know are secure, and then around the edges maybe you need some net new code, but mainly you're just pulling things in, that could be huge savings for the company in terms of their AI overhead.
LB Exactly. And I think it's more green and sustainable, if you think about it in the long run, in compute power and in storage and everything. Just something I'm playing with.
RD All right. I like it. It's out in the wild, other people will interact with it. You've open sourced the idea, so I love it.
LB Yeah. Whoever's up for it.
[music plays]
RD All right, thank you folks for listening. It's that time of the show again where we shout out somebody who came on Stack Overflow, dropped some knowledge, shared some curiosity, and added to the community. Today we're shouting out a Lifeboat Badge: somebody who found a question of negative three or less and dropped an answer that got a score of 20 or more. Today we're shouting out friend of the show contributor Charlie Martin for dropping an answer on: “How does this proof, that the halting problem is undecidable, work?” If you are curious, Charlie has an answer for you. I am Ryan Donovan. I edit the blog, host the podcast here at Stack Overflow. If you want to reach out to us with comments, complaints, ideas, et cetera, you can email us at podcast@stackoverflow.com. And if you want to reach out to me directly, I'm on LinkedIn.
BP Hello, everybody, I am Ben Popper. I'm one of the hosts here at the Stack Overflow Podcast. If you want to hit me up, I am @BenPopper on X, or you can shoot an email over to the podcast email here at Stack Overflow and we can discuss what you want to hear in the future. Laly, tell them your whole name, your title, where you want to be found online if you want to be found, and then a call to action to go check out something about Bit if there's a place where developers can go to learn.
LB Thanks so much for having me here. It's been a pleasure. My name is Laly Bar-Ilan. I'm Chief Scientist at Bit.cloud. I'm on LinkedIn, Laly Bar-Ilan, and pay us a visit at bit.cloud.
RD Thank you very much everybody, and we'll talk to you next time.
[outro music plays]