Ryan and Ben welcome Tulsee Doshi and Logan Kilpatrick from Google's DeepMind to discuss the advanced capabilities of the new Gemini 2.5, the importance of feedback loops for model improvement and reducing hallucinations, the necessity of great data for advancements, and enhancing developer experience through tool integration.
Episode notes:
Gemini 2.5 is DeepMind’s most advanced model yet, with strong reasoning and coding capabilities.
Connect with Tulsee on LinkedIn.
Connect with Logan on LinkedIn and Stack Overflow.
Check out our previous episode with Logan. We discussed his unique path from coding to AI to product, the challenges of non-determinism in AI models, and surprising lessons from working at the Apple Store.
Congrats to Populist badge winner Pascal MARTIN for answering the question PHP echo vs PHP short echo tags.
Our 2025 Developer Survey is live! We want to know what your developer life is like!
TRANSCRIPT
[Intro music]
RYAN DONOVAN: The 15th annual Stack Overflow Developer Survey is live, and we wanna hear from you. Every voice matters, so share your experience and help us tell the world about technologies and trends that matter to the developer community. Take the 2025 Developer Survey now. Link is in the show notes. Hello everyone and welcome to the Stack Overflow Podcast, a place to talk all things software and technology.
I am Ryan Donovan. I edit the blog, host the podcast here at Stack Overflow, and I'm joined as co-host today from the notorious Ben Popper.
BEN POPPER: Hello everybody. How's it going? Ryan, we're lucky today to have some folks on from DeepMind Gemini to chat with us, not just about sort of like, okay, here's what it can do, but also how does this stuff get built? Like what creates these improvements? What unlocks new capabilities, which I think is, you know, always been what interests the Stack Overflow audience.
RYAN DONOVAN: Yeah, so we have two folks from Google's DeepMind talking about the current number one model, Gemini. Tulsee Doshi, Senior Product Director, and Logan Kilpatrick, Senior Product Manager. So welcome to the show.
LOGAN KILPATRICK: Thank you for having us. This is round two, Ben.
BEN POPPER: Yeah, Logan and I have chatted before. I know you mentioned that you're a bit on sort of like the AI studio side. Tulsee, we've never met before. You're a bit more on the model side. Can each of you perhaps define a little bit of what is your focus and scope of work? What is an evolving focus overall for Google DeepMind's business?
TULSEE DOSHI: First all, yes, thank you for having us and I lead the product team for Gemini Modeling. So basically, what are the capabilities we wanna build into our model? What is our model roadmap? How do we make Gemini the best model for our users internally and for developers and customers externally? So focused on our core development there.
LOGAN KILPATRICK: Yeah, and Tulsee’s team makes the best models and then our team gets to take the models and find a way to sort of bring them to the developer ecosystem. So we do that through AI Studio, which is sort of the front-end product experience, and then underneath the UI and what people actually build with is the Gemini API. So that's where developers take Gemini and hopefully build it into their products.
RYAN DONOVAN: Today we're talking about the new Gemini 2.5. Is it experimental still, or is it gone Pro?
TULSEE DOSHI: They're preview models, but what that actually means for both Pro and Flash is that you can build on them. They're priced models, you can scale them up and use them. Really what preview means is that we are still testing these models and so we expect there to be new versions and iterations of the models that we'll release in short order before we stabilize a model to being around for a longer period of time.
LOGAN KILPATRICK: This difference matters a lot for developers. There's a lot of people who need models to not change just because of whatever the industry that they're in, and I think the 2.5 models right now are going to change. We have not finalized them. So that's sort of the persona that you're in right now. It's like you can definitely do the testing, but if you need them to be consistent, we haven't yet gotten to that point.
TULSEE DOSHI: And part of that is really just to also be able to get feedback. Right? So we want these models to be in the hands of users. We want to be able to understand what's working and not working, and then take that feedback and then apply that to a model that maybe doesn't change as much over the course of a longer period of time.
BEN POPPER: One of the things that I heard a lot about when Gemini 2.5 arrived was on the pricing side, which does matter a lot to developers. You know, how much tokens input and output cost, and especially if they're working with a big code base or trying to go through a lot of iterations. Can you talk a little bit about that side of the curve? It does feel like the scaling laws initially were focused on how much smarter can it get with each turn of the wheel. And as you know, we get up into the 97, 98, 99%, it's slowed down. We're like eking out those last gains of genius at every subject, but the cost and maybe the size in terms of the ability to like fit on a phone, you know, some of the scaling there seems to still be really impressive. Is that something you think about on the product roadmap, the cost for the end user or developer?
TULSEE DOSHI: I'm glad you asked because we talk a lot about this idea of like a Pareto frontier of cost to quality. And one of the goals we set for ourselves on the Gemini models is we want to be at the top of that Pareto frontier.
So you can think about it as how do we maximize the balance between best cost for quality? So it's not just about the quality benchmark itself, it's about the combination of cost and quality and how these two things play together and we want all of our models to sit at that frontier. And so that's actually a very, very strong goal for what we build.
And actually with 2.5 Flash, which we most recently released, a big priority was actually giving developers control over that dynamic. So being able to turn off thinking, for example, or being able to control how much the model thinks with different budget actually allows developers to have control over cost and latency for different quality thresholds. And I think that's actually super important to the developer experience and super important to what we want to build, which is the most usable models, right, that are also incidentally the best.
BEN POPPER: Fascinating to think about setting different parameters so similar to sort of the evolution of cloud computing, where it'd be like, look, I want this, you know, workload to run at night or, you know, I would like to achieve this, but I can't let costs get out of control. So if it gets to here, send a warning or slow things down. And to now see that in AI workflows is really interesting.
RYAN DONOVAN: I heard of a way of sort of reducing costs called Chain of Drafts. It's a sort of chain of thought prompting where it'll come up with a prompt and then it'll reduce it, so you're actually outputting less tokens in the end. Is that something you've explored at all or heard about?
LOGAN KILPATRICK: One of the things with the 2.5 Flash model launch was we introduced thinking budgets, which is like perhaps not exactly what you're describing, Ryan, but is like a similar vein. How do you give developers that level of control to say we don't want the model to think as long as it possibly can to solve some problem, and we don't yet have this on 2.5 Pro. The 2.5 Pro model, for lack of a better word, thinks as much as it thinks it needs to, but we want that level of control for developers on Pro as well. And hopefully we'll get there soon.
TULSEE DOSHI: And then I think to your question about Chain of Drafts and kind of different algorithmic approaches that we could be taking, I think one area that is super critical for us is thinking efficiency generally.
And, and how do we actually train the model appropriately to think the right amount? Because there is something about emitting less tokens, but it's also about how do we preserve that right quality balance in doing so. And so from just like my brief understanding Chain of Drafts is one methodology to do. So, I think we're exploring many.
BEN POPPER: We were chatting with someone recently who was one of the original authors of a RAG paper out of Facebook research, and they were making the point that we often talk about, hey, how close can we get to eliminating hallucinations? But actually as Chain of Thought gets longer and longer and thinking goes on for more time, you may get more hallucinations as some of the recent papers for newer models have shown. It doesn't necessarily always go down. I mean, I thought that was really interesting that maybe, as you said, you have to budget these different capabilities and these different concerns within the model's architecture and within its inference time workloads.
TULSEE DOSHI: Yeah. I think the other thing to think about there too is making sure that we're thinking through all of the different areas that we're trying to hill climb against, right? So when we evaluate our models, we're evaluating them on the reasoning metrics. We're evaluating them on code metrics. We're also evaluating them on factuality metrics to try to understand how these different things play together. And so, you know, there's definitely a number of different trade-offs every time you're hill climbing a new metric every time you're pushing for a new frontier. So better understanding those, I think is super important when we're actually choosing a final model to release in terms of how we think through all those balances.
LOGAN KILPATRICK: Yeah, and I think there's two other quick dimensions to this, and Tulsee jump in if I'm misrepresenting this here, but I think one of them is like tools being a mechanism to stop model hallucinations because there's just things we know the models are not good at. Like they're never gonna have the most up-to-date information from the internet. Therefore, like don't even pretend that you're gonna be able to guess what the weather's like in Chicago right now. It's not possible for the model. So you can use search to do that. A bunch of the math stuff with code execution, I think, solves like a whole class of really gnarly hallucinations.
But I also think the reasoning step is super important and from my understanding what we've seen is the model is able to catch cases where it's like making blatant mistakes that historically it wouldn't otherwise have been able to because it has that self-reflection process as it's coming up with an answer. It is interesting that it feels like generally over time as the models reason more and have access to more tools, hypothetically, you'd expect the number of hallucinations to continue to go down because the model, it's not like stuck with the hallucination, whereas like in the previous iteration where like you basically get one shot to answer the question, if you go down the wrong sort of chain of thought, you end up putting out the wrong answer and now you can sort of self-correct during that process.
TULSEE DOSHI: I think this is where, to Logan's point, the tool calling piece becomes so interesting because if you can then call search as a tool, you can actually like create ways to validate and verify information in a way that can actually be more grounded and therefore reduce hallucinations. And so I think there's also a world we're moving to where we also need to think about the full system that is the model, and not just the individual model itself, but how it works in conjunction with tools and in conjunction with how it is actually deployed.
When you look at metrics like SWE-bench for example, they measure not just the model, but the model and the harness in which it is being deployed and so I think all those combinations of factors are gonna be super important.
RYAN DONOVAN: I wanted to back up a little bit because you hear a lot of people talk about reasoning models, and I want to sort of understand what is the model actually doing in that case and what is Gemini specifically doing?
TULSEE DOSHI: We've been talking about our models at Gemini 2.5 series as thinking models that are models that actually produce a set of thought before they respond. Right? And so to something Logan said earlier, the idea behind these reasoning capabilities is that the model can actually form a plan. It can sort of reason through that plan, it can critique or verify before it actually produces an output.
And so you see that significantly improving quality in areas, especially where that kind of planning is valuable. That's why you see such improvements in math or in code where that kind of reasoning process can significantly improve an output. You can also see that improve even in creative writing as the model can kind of reason through how it will pursue that task and then actually pursue those step by step before it produces the final response.
And so I do think that inference time approach of being able to take the model and then leverage that additional inference to produce the final outcome is a direction that we're seeing kind of across the industry and one that we're super excited about for Gemini in pushing forward a lot of these like key use cases.
BEN POPPER: You mentioned tool calling and that being a solution to, for example, intractable problems like the number of r’s in strawberry. Nobody can tell you the number of r’s in strawberry. It's just impossible. When the model is going through these different steps and iterating or reflecting, is there more of an old school, like not LLM based or you know, non-deterministic rule, but like a set of rules that you write in that's like, this is the time to make a tool call, or this is the situation where you would call a tool? Like how does the model know when to stop reasoning and make a tool call, I guess.
TULSEE DOSHI: We do a lot of this right now in the way that we train the model, right? So giving the model examples of successful tool calls or scenarios in which a tool call makes sense or working with the model to train it on, here's a certain amount of thinking that you should do, and then here's an answer. And so we don't have deterministic rules that say, for example, if this then always call tool because I think we very much want to invest in training the model to understand these nuances.
And also because I think often these types of rules can lead to a lot of collateral damage because if you make them sort of across the board, there might actually be cases where you call a tool that doesn't actually make sense because you're overusing that strength.
The other thing is we want the models to work well, both with, for example, tools on and tools off. So we are training the model in both of those scenarios to make sure that if you ask the model questions when the tools are on, it knows how to use those tools well. But if you also ask the models questions when tools are off, that you will still get a very strong experience.
LOGAN KILPATRICK: This is also where you wanna have the opportunity to interject developer control. It's like, so just having the tool– knowing that the model has access to the tool. Yes, the model's been trained in some cases, but like that's, you know, to a certain extent, that's like our worldview of like when the model should be using those tool. That worldview like perhaps doesn't sort of carry across all developer use cases.
You maybe wanted to only interject that tool in a very specific case and developers have that level of control and they can sort of guide the model with system instructions and other things to make sure that the model's not just like, you know, calling search every time you ask some very generic question about some updated information, maybe there's another tool that you actually have that's not our search tool that would enable you to get that answer. And it's super important that we give developers those knobs to make the decisions.
RYAN DONOVAN: When you talk about tools, you mentioned search. What are sort of other ones and is this all a sort of like inference-time data addition?
TULSEE DOSHI: So, when we talk about tools right now in AI studio or in the API, you can access, search as a tool and so grounding abilities, you can also get code execution, right? So these are two tools that we've sort of natively built in to the model and really invested in high performance there. We also want developers to be able to call functions more broadly, right? So we've been investing in function calling as a more generic property of the model, if you will, to be able to actually call any function that you want to be able to have the flexibility to call. And you can see in the Gemini app a much more robust set of tools that power that consumer experience, whether that's calling maps or you know, other parts of the experience, workspace, and Gmail, and docs.
Being able to actually interact with the world around you, I think is super important. So, and when we think about tools, I think you asked about inference. We want improve tool calling throughout the model training. So it's not just about the inference time, it's also about training the model with fine-tuning data and RL to actually make sure that the model is good at calling these tools and understands how to do that effectively. And then inference time efforts can build on top of that.
LOGAN KILPATRICK: And we have lots of discussion internally about like what are the right set of hosted tools for us? And you can think about like search and code execution as hosted tools that Google is providing for developers versus what do we just need developers to do themselves?
And like there's a bunch of other hopefully interesting areas where it would make sense for Google to host those tools and like searches an obvious example of this, like we have this really great world-class search infrastructure, we could bring that to developers, it's unique, it's differentiated.
And code execution's another one where like that infrastructure is, is actually not trivial for a developer to stand up themselves, but there's a whole lot of things where like it does make sense for developers to go set something up themselves because there's a bunch of great, you know, third party APIs that do whatever that thing is. So hopefully we'll see like a nice balance of both of these things in the future.
BEN POPPER: When I was getting into vibe coding, you know, early on in 2023, often where I would get stuck is the connection between the front-end and the backend. The API key, like you said, the tissue that goes between different services. Now I play around with the Lovables and Bolts of the world and, like you said, they build in these connectors for you. Okay, you want a backend, just click this button. You'll have Supabase, you know, more recently I was using. Okay, you want a payment stuff just, you know, drop the key in here and Stripe will be set up, right?
So they understand that there may be things you wanna do. That building from scratch would be a waste of your time. And it's really interesting to think, Logan, to your point, right, about that you want authentication? Google can do that for you, you know? You want to have a search bar in here, you know, we can do that for you. So, not just, right, code generation, but tapping into the capabilities that are in the model provider’s entire wheelhouse of capability.
LOGAN KILPATRICK: One of the interesting threads that this pulls on is just around like, what is our place in the ecosystem from like what we build and like how opinionated are we and like how high up the stack do we go versus like, do we focus on primitives, all that type of stuff? And I think from the developer ecosystem, it's pretty clear people want model agnostic stuff, in a lot of cases. Developers are happy to buy into model like provider services, in some cases. And it's this like really interesting balance as you think about like what do you build for developers so that developers still feel like they have the control and they don't get locked into whatever the product or service is, but it does introduce lots of problems.
I was having a conversation with someone this morning about the live API stuff is like a great example. There are standards for things like normal chat conversations and it's a pretty similar infrastructure across all models in the ecosystem. As you start to look at some of the like new frontier stuff, it's all completely different.
It's going be really interesting to see what happens with that and like live API is a great example. We have this incredibly unique thing that not a lot of other people have, and we're also seeing a ton of adoption and momentum around that. So it's very interesting.
RYAN DONOVAN: I wanted to ask about the native multimodality. You know, when you talk about embedding, it’s like you can embed anything, right? Do you have the sort of cross meaning like is it possible to have the same embedding or similar embeddings for, you know, a picture as a word?
LOGAN KILPATRICK: It’s definitely something developers are asking for. Google Cloud offers a bunch of different embedding models. We have a Gemini embedding model that today does just text, and there's definitely a ton of interest in like a single embedding model that could do all modalities.
TULSEE DOSHI: I think it's a great question and it's definitely an area we're leaning into. I think one thing we do really believe, especially if you talk about embedding specifically, is that there is a lot of value in multimodality and actually combining the world of like, for example, text and imagery and being able to produce embeddings that can look across of these, so that that is something I think we fundamentally hear from developers and also just believe is a good direction to be traveling in.
RYAN DONOVAN: When you say native multimodality, what do you mean?
TULSEE DOSHI: That's a great question, and maybe we were talking about two separate things there too. So one is when you talk about embedding specifically, there's both embeddings in terms of how we actually build the Gemini models and embeddings that power them.
There's also embeddings that we actually release as an offering built on Gemini, so we are working on multimodality in the context of both of these things. For the Gemini 2.5 models, we do consider them in many cases, to be natively multimodal. And I think we think about that both in the input context and the output context, right?
So in the input context, we want Gemini 2.5 to be able to understand any form of media. So whether you ask Gemini something in text or video or audio or image, Gemini should be able to do amazing things with that. And actually, like one thing that's really awesome about the 2.5 models is how good they are at video understanding.
So you can give these models a video and then ask actually based on that video to generate a web app and you'll see really interesting outcomes. Or you can give a video and ask to caption or timestamp that video. And Gemini will do a pretty awesome job of breaking that down. And so I think that part is super important to us.
Then there's native multimodality in the output. So can Gemini generate images or can Gemini generate audio, which actually Gemini natively generates images. And that's a big part of building out native multimodality is being able to combine these aspects together. And what that allows you to do is more nuanced things in the generation of the content, because you benefit from Gemini's world knowledge, right, in the way that you're actually generating the media.
BEN POPPER: So one of the things that always stood out to me, in which I've sort of picked around the edges up, but never gotten like a great answer, is why the process of creating a new model was thought of as being really expensive, compute intensive, and delicate. You know, the kind of thing where like, okay, the data center goes out over here, we've got to start all over again. It feels like these days, like you were saying. Every time a model comes out, there's a lot of different flavors of it. Maybe we'll even go backwards in time and we had 4.5, but now we'll have 4.1, which is better for developers, you know? So it's like, you know, there's different flavors, like the same model.
You had mentioned this at the beginning, like, okay, it's out now, is we're still experimenting, we'll still learning, and we'll solidify this eventually into like Gemini 2.5 final. So like what allows you to do that? Is this, we're just sort of fine-tuning, we have new techniques, we're actually putting out different models every time we change the number.
I'm just curious because it used to feel like we were waiting for that big drop 1, 2, I'm waiting for 3, or 3 to 4.5. Huge difference. But now it feels like people are constantly releasing tweaked versions of the model that, to your point, may suit one group's needs or show a preference for being exceptional in a certain area.
TULSEE DOSHI: From our perspective, there's multiple stages to building a model. So you can think about it as there's pre-training where you're training that base model. There's post-training where you kind of iterate on top of that and fine-tune it further. There's inference time that you can build on top of that, and all of these stages lead to different types of improvements in the model and then you may want to tweak any of these stages to make the model better.
So I think one thing that we're really excited about is A, how these stages have come together to build 2.5, but then also what that allows us to do is at any of these stages, say, hey, here's maybe something we're seeing in terms of feedback. Here's the best way to improve it.
The improvement of these flywheels and being able to speed up the process in which we can improve the model, I think has been true across the board for us. And then that actually allows us to then keep shipping, improve versions of the models at a quicker pace. And that just, I think, allows for a better feedback loop with developers, because otherwise, if you treat these as like very stagnant, kind of separate multi-month independent efforts, you don't actually get that ability to kind of iterate quickly in a way that is useful for the end user.
LOGAN KILPATRICK: Just to add to this, I think maybe the subtext of this story that Tulsee is describing is just like the amount of investment and innovation that's happened in the actual model training process.
If you look back even like two years ago or even a year ago, the cadence of models was different because people had yet to build the infrastructure that made it so that you could build new models very quickly and like actually get them deployed and get the feedback from users and like there wasn't enough compute to do that. All of those things didn't exist.
And I think the last three years of the entire ecosystem building out that infrastructure and making it so that you can be more nimble and answer more of these questions based on real world developer feedback in our context and user feedback, I think has been super positive.
Tulsee, you and I have had this conversation a lot about like, what is the outcome of us doing all of these experimental launches and new preview launches and stuff like that. And the bar that we ask developers to hold us to through that process is, we're not doing this for the sake of just putting out more stuff. We're doing this for the sake of making better models for you and ideally making better models for you and getting them into your hands so that you can actually deploy them into production faster than before.
That's the bar, and if we don't meet that bar, we'd love people to hold us accountable, but I do think 2.5 has been a story of us hitting that bar in a lot of ways, which I've been really, really happy about.
BEN POPPER: I think I had a paradigm in my head that the initial stage, that pre-training stage was a month long process you had to commit to that would cost tens and then people would project hundreds of millions of dollars, and so I didn't imagine that that was something that you could return to when you were sort of figuring out where you want the final version of something to settle. I don't wanna dig into that deeper, but just interesting to think that all along that process now you can be just so much more nimble.
RYAN DONOVAN: And I know y'all had mentioned before there was a bunch of research that went into the improvements of this model. Can you talk about what the research was and how it affected the model?
TULSEE DOSHI: Actually, I'll maybe speak to something that I think is particularly cool about the models that we shipped. One is to this pull point of like the different model stages, I think one of the things that's really impressive about the 2.5 models is for us at least, how much these stages have built on top of each other and work together.
And I think this is also about when you're doing pre-training, actually training the model to also be more amenable to changes in post-training. When you're doing post-training, actually bringing that inference time into the way that we're post-training so that the model also responds to that appropriately.
And so, I think, if you think about these stages as fully independent, that can lead, I think, to less of that one plus one equals three type effort versus actually training these to, to be sort of building on top of one another. So I think that's one. Even just a year ago when you're talking about thinking models, a lot of the focus for these models has been on these academic benchmarks and on reasoning. And another big thing that we invested on in this model is not only hill climbing the academic benchmarks. But also hill climbing user preferences and you know, vibes, if you will, and really trying to make the model actually enjoyable to engage with.
And so I think those are actually two things that we've been parallel tracking as we've thought about the fine-tuning as we've thought about the RL is, is really making sure that we're not just building on academic skill, that we're building on both of these and bringing them together well.
BEN POPPER: I love that point because it feels like for a while what we've been seeing is like now it's a silver medalist in the math Olympia. Now it's a gold medalist in the, you know, code challenge. Now it's a PhD level in 19 different subjects. And I struggle to imagine the person who needs all of those capabilities, right?
And so now I think we're getting much more to an era of like the tooling, the UX, the product con– the connectivity, the tool, whole product suite. Then Chelsea, to your point, the, the vibes, you know, people talk about this a lot and they're like, I don't like this one. It's so boisterous. It just goes on and on.
It's so pedantic and like, why can't it just give me, you know, like answer as if it was talking to a friend. So the idea that the models are so smart now that we have focus on vibes separate from vibe coding, this is the model's personality and style almost, is really fascinating to me.
LOGAN KILPATRICK: To add another dimension to this, which I think ties to this vibe story, Tulsee, you sort of represented the foundational research work that's happened. It's also interesting that like in parallel to all of that foundational research happening, every time you release a new model with these capabilities, it kicks off this ecosystem wide applied research story. And this is open research, applied research questions as to like, I'm a developer, how do I know what the right thinking budget is to set for this model?
It's very interesting, this intersection between applied research and product development and the line is very blurry now, and every time the new model comes, there's a new frontier of you could actually have a differentiated product advantage if you can figure that thing out before your competitors can.
And these are some of the questions that we don't have all the answers to either. That's why we want to release models to the world, is because you can only answer a lot of these things when people actually use them and build real product and sort of put them through the paces which has been very, I think it's fun. It's just cool to see this happen in practice and there's lots of interesting stuff to figure out.
RYAN DONOVAN: I want to ask a little bit about the coding aspect of it for the developers. We talked to a lot of folks putting out AI generated code and you know, I don't think anybody really trusts it yet. We've seen some vibe coding horror stories on Twitter and that, and I wonder what sort of improvements and things you're thinking about in terms of creating code.
TULSEE DOSHI: I mean, I can speak to the model side, but I think actually a big part of this is also the developer experience. To your point, the models are getting much, much better at code. They're not infallible. And so how do we actually help you build the best companion code product is I think a lot of what we're thinking about. And so a lot of it is really trying to identify these use cases, whether that's editing code, transforming code, you know, allowing you to like inspect and ask questions of code. I think all of that's super important, right? So being able to build a partner between you and the model, I think is actually a really critical part of like how we A, just continue to improve these models and also how we can actually provide real value to developers in that way.
And so for us, a lot of that is about thinking through what are the right evals, what are the right use cases, what are the right areas that we really want the model to shine in when we're thinking about code development and providing that to developers in a way that they can continue to build and iterate on as opposed to thinking about it as fully end-to-end.
But I do think there are some areas where we are seeing that end-to-end kind of one shot performance start really shining like end-to-end web app development where you can just with a single prompt build something awesome. And this goes Ben, to your example of vibe coding with Lovable and with Bolt. And I think this is gonna be about the combination of the model and the tooling that we build around the model to make that whole kind of end-to-end flow work well.
LOGAN KILPATRICK: There's a bunch of angles on this. The vibe coding horror stories, I think, a lot of them actually go back to this like very fundamental AI problem, which is if you don't give the model context, it's going to make a bunch of assumptions and it's likely that those assumptions are not going to be in the same order of magnitude or class of assumptions that you are going to make. And in a lot of cases, actually, that's why, you know, the vibe coding products out there can sort of provide some of this scaffolding. Like here are some frameworks that are scalable and secure and authentic, and maybe the model would've chosen to do those. But like as a developer, as someone vibe coding who doesn't know those different tools and technologies, it's unlikely you're going to make the right decision.
So I think the infrastructure that those products are providing is really important. But also separately that– the one end of the extreme where you just wanna zero shot, build those products. I think on the other hand is like the enterprise use case of this, of like I have a large code base. And in those contexts it's actually really interesting and I was talking to someone yesterday who was telling me about like how much success they're seeing for that particular use case.
Because companies actually have really well articulated style guides and they have best practices and they have engineering, how they build code and deploy software, et cetera, et cetera. And all of that is like the perfect juicy ripe context to give to the model to make it actually helpful for you. And without a lot of that context, you are kind of just taking a shot in the dark and like maybe you don't end up with what you want. So it is interesting to see both ends of the spectrum working, but in a very different way.
BEN POPPER: So to your point about what really enriches the model, there is all that extra context, all those, you know, extra instructions and all this stuff that is internal wisdom to a certain company. There was a discussion recently about training for the next big number release and I heard something really interesting was like the paradigm shift is that we're no longer in the regime of being compute limited. We're in the regime of being data limited and what we're really looking for is great data that we haven't accessed before or a new way to do evals to understand when we're improving or, more to your point, to combine what we have with the data that's inside of an enterprise, whether that's a really big code base or some great documentation or architecture diagrams. Do you, either of you, have an opinion on whether or not there has been sort of a regime change or if data is now as if not more important than compute when it comes to what might be the limiting factor for improvements?
TULSEE DOSHI: It's a good question. I think data has always been critical and I think continues to be super critical. So, I do think it is true that high quality data is one of the most valuable things you can do to improve model quality. And I think you also touched on something else, Ben, which is evals, Which is a form of, of data. Right?
But I think actually is to me, so, so critical. Right? Especially as we keep thinking about what is this next frontier of quality? What do we want to build? One of the most valuable things we can do for the team is really bringing the right evals and the right story of what we want to hill climb towards because that points us in a direction. It also gives us a north star. It also like helps us continue to push the frontier. And so I think that that story is a really critical part of I think investing in the model development process.
LOGAN KILPATRICK: If you look at a lot of the work that was happening in the open source ecosystem for probably like the middle of 2024, roughly, most of like the open source innovation story was what happens when you do fine-tunes or large scale model training processes on open-source models making data modifications. It was like minimal amount of compute possible. How do you sort of make the model significantly better just by training, switching up the open-source model data mixture and like lots of really, really interesting work highlighting this point that like if you have really great models with really great data, it actually skews the model performance in a really significant way.
TULSEE DOSHI: I think what it really speaks to is, as we start thinking more about thinking and as we're trying to scale up, you know, RL data is just such a critical part of that, and it's, the interesting thing is it's also more about quality, I feel like, even than quantity. So it's how do you find high quality data that really represents the kind of task and use case that you want to build out? And that becomes also, I think, for a product manager, like a very fun challenge of like, what does that high quality data look like?
LOGAN KILPATRICK: And it's interesting because a lot of that thinking data doesn't exist, which is very unique. Like if you just think about like how often other than like people's journals, like how often do you like actually as a human like yeah describe your thought process or why you're doing something. Like so much of like the artifact of human work is like this final end deliverable and the expectation is you've done all the thinking behind the scenes and you actually don't wanna expose that stuff to like whoever your stakeholder is, if you're creating some report or some deliverable.
And it turns out for the models to be able to replicate a lot of that work, they need to have some insight into the process that led to that. And there's so few domains where I think it's clear that there's like a lot of already existing data in that ecosystem.
BEN POPPER: Well, you have to be brave and step up for the neural implant and, it'll see how you're thinking about it and then it'll be as smart as you.
TULSEE DOSHI: (laughs)
LOGAN KILPATRICK: Easy peasy.
RYAN DONOVAN: Alright, well thank you very much everyone for listening. It's that time of the show where we shout out somebody who came onto Stack Overflow, dropped a little knowledge, shared a little curiosity, helped out. Today we're shouting out the winner of a populous badge, somebody who came on and dropped an answer on a question that was so good, it outscored the accepted answer. So congrats to Pascal Martin for their answer on “PHP echo verse PHP short echo tags.” We'll put that in the show notes if you're curious.
I'm Ryan Donovan. I edit the blog, host the podcast here at Stack Overflow. If you have feedback, comments, concerns, email us at podcast@stackoverflow.com. And if you wanna reach out to me directly, you can find me on LinkedIn.
BEN POPPER: Hey everybody. I'm Ben Popper. I'm a host of the Stack Overflow podcast. Find me on LinkedIn at Ben Popper. We can chat.
LOGAN KILPATRICK: My name is Logan Kilpatrick. I am all over the internet including Stack Overflow. Go– don't upvote my stuff, but go check out all the questions.
TULSEE DOSHI: I'm Tulsee Doshi, and I lead the product team for Gemini at Google, and you can find me less prevalently than Logan on the internet at LinkedIn or on X.
RYAN DONOVAN: Well, thank you very much everyone, and we'll talk to you next time.
[Outro music]