Will AI fundamentally change software development or just add some efficiencies around the edges? Surveys from Stack Overflow and Github find north of 70% have probably already tried using it and many incorporate it into their daily work through a helper in the IDE.
It's also worth reflecting a bit on the technology sectors that didn't have as great a 2023: crypto, VR, and quantum computing still seem far from mainstream adoption.
We dive a little into the half-life of skills, which seem to be shrinking, especially in IT. Got any resolutions to learn something new this year?
And what about the data we use for training? We highlight a comment from Kian Katanforoosh, a lecturer who helped create Stanford's Deep Learning course with Andrew Ng, who says we'll run out of high quality data as soon as 2030.
A big thanks and congrats to Stack Overflow user Corn3lius for helping to answer a question and being awarded a life boat badge: How can I create spoiler text?
TRANSCRIPT
[intro music plays]
Ben Popper Transform your business with Microsoft Azure. Easily move workloads with Azure Migrate and Modernize, and build new AI solutions with Azure Innovate. Access comprehensive resources, expert assistance, and cost optimization tools. Visit aka.ms/azureheroofferings to learn more. Make sure to use that link to let them know the podcast sent you and support the show.
BP Hello, everybody. Welcome back to the Stack Overflow Podcast. You are listening to this, I hope it is the year 2024. I hope we all made it and you had a nice holiday and welcome to a new year. I am Ben Popper, Director of Content here at Stack Overflow, joined by my partner in crime, Ryan Donovan, the Editor of our blog and maestro of our newsletter. Ryan, we're doing the last episode. We're two days away. It's Thursday now, Friday is the last day before our big break. It feels good.
Ryan Donovan That's right. Got bells jingling out there.
BP Bells are jingling. So I thought we would do a little year in review of the podcast. Some big things happened this year.
RD You could say that.
BP It feels like a momentous year in the history of software development. I was looking back at things just trying to get a sense of the history as we were working on a piece, and the transformer, which is the architecture that has supercharged neural networks and helped so many folks make so much progress– not the only thing, but one of the most important things –was created in 2017. And then a little company nobody had really heard of or thought too much about called OpenAI released GPT-1 in 2018, and most people didn't care. It didn't really make much noise. Some people noticed, whatever. It was a thing, but it wasn't a big deal. And then in 2019, they released GPT-2 again. I think I remember each time you'd see some tweets, some people would show things off, you might play around with it a little, I remember, but it didn't become something that's like, “Oh, well, this is going to change the way we work,” or, “This is revolutionary technology.”
RD I saw an app that was AI Dungeon. You could go on a text adventure that had anything. I was like, “Oh, that's cool.”
BP You could make a little dungeon crawl, you could write a short story, but often it would go off the rails pretty quick. And it certainly wasn't answering questions. It was like, “Start a poem and I'll finish it,” or, “Start a story and I'll finish it for you.” So then in November of 2022, ChatGPT comes out. I think you and I were already playing around with some things that felt kind of revolutionary a little before that, which is Midjourney and Stable Diffusion. You were running your own instance. So Gen AI, on the image side, I think people were already starting to feel like, “Wow, this is pretty world changing.” It didn't capture the world's attention, I don't think. It didn't go viral, but it was clearly something where we were talking to ourselves– “Are we going to be hiring illustrators going forward? Are we going to need an art department? How are things going to change?”
RD I think it sort of goes to show that you can have very cool technology, but until you have a good app or interface to it, nobody's going to jump on it.
BP Right, I like that. So that's important because one of the things I think that made ChatGPT a success– and maybe if it hadn't been, we wouldn't have made this turn this year, we would have made it in a different year. Well, first of all, it got bigger. Every time it got an order of magnitude bigger and better just because they were able to. But then they spent a lot of time afterwards investing in reinforcement learning with human feedback. So they fine-tuned the model basically to be a great chat partner, and I think that paid a lot of dividends as well; that it wasn't the raw model by any means, that it was a model that had been optimized to be your chat buddy.
RD And having that human in the loop has been sort of key to AI up to this point. A lot of people would train AI on human tagged data sets, going through things like Mechanical Turk to make sure that they had human input on what was a frog and what was not a frog.
BP Right. And so coding assistants inside the IDE were already out but have become more and more prevalent, and if you believe the statistics put out by the folks who run some of the largest IDEs or code repositories, upwards now of 70% of code that's being committed has some AI generated code in it. Whether that's just an autocomplete at the end or a full line or a full function, we don't know, but it's becoming ubiquitous, I would argue. What do you think?
RD I think it's just getting easier just to do the little stuff. But in some of our pieces, especially some of the recent ones I've been seeing, a lot of folks now that they've had time to play with the AI code generators, saying it's not actually the productivity hack that it was. It's still wrong a lot of times. It still takes a lot of time to get the prompt right. So if you're doing just the dumb stuff, type definitions, whatever, then it works fine. But for the big things, maybe less so.
BP I like what you're saying, and I think there's a little bit more to it. We had a great episode with Matt Biilmann, CEO of Netlify, who's obviously very tuned in to how a lot of developers are working, and he said that they use it to– it was something where they would come back to it every time they wrote code. It wasn't the code itself, but it was something adjacent to it. And similarly, Ben of Dev.to was like, “I use it to write tests. It's good enough for that.” So like you said, maybe it's not writing the code that's going to commit, that's going to prod, that's out there, but maybe it's good enough to handle your code comments or write some of the tests for you, which is a nice productivity hack in that sense.
RD This has been the year of AI. If folks in the tech industry aren't saying that's the sort of overall theme then they are in a niche area.
BP Yeah, you're probably sick of hearing us talk about this. What are we going to do?
RD It's the new hotness and there's a lot of possibilities there. I think people, as they're exploring the possibilities and narrowing down what it can and can't do, some of the hype, some of the shine is going off of it, but then it's becoming a tool that actually people can use and use in specific ways.
BP Right. Here it is– I mentioned this to Matt Biilmann and he said they use it and they open source it to make Storybook components. You make a new component, now you've got to make the Storybook different examples, so now it just does it for you. You wrote it and then it’ll give you a Storybook story with the components. So that's how they use it and he says that it's totally accelerated how they work, which is a good thing. But within that, security risk, privacy risk, maybe not letting it too close to the core of what you're doing, keeping it on the periphery, writing the test, writing the Storybook components, stuff like that. Let's talk about two things that didn't break through or that had a bad year.
RD Love it.
BP Crypto/blockchain had a rough year. I'm not going to write it off forever. There are still true believers. If somebody creates a blockchain application or service that is helping people, maybe that's cross-border transfers, more power to them, but obviously a very rough year to see the largest players collapse and a lot of the business turned out to be fraud and a huge amount of developer energy went into that for a long time, and so that's now shifted to new areas.
RD Well I think with that one, it never had that killer app. It never had the ChatGPT to have it break through to, “Oh, this is what you use it for.”
BP Correct. Good point.
RD And I was never a huge fan of it just because I think the financial end of it always seemed like it was based on speculation or getting around governmental control of money.
BP It had the opposite of a killer app. I was always like, “Now it's going to take two or three steps and be a little bit slower and more expensive to transfer fake money.”
RD Its killer app was ransomware.
BP Right, it's very good at that. We shouldn't say it's not being deployed regularly in the real world to do things. It's back to its Silk Road roots. All right, and then the other thing I would say is quantum computing. This is an area where we have people on the show once or twice a year. It's always super fascinating. They talk about how they're doing demos and trying things in the real world, but I think we've yet to see it break through to something that exists outside of a laboratory. It hasn't become part of anybody's workload or tools or everyday, and certainly I don't think has captured the imagination of the mainstream.
RD There was that show Devs that sort of took a swing at having quantum computing in a realistic and speculative manner. Realistic, maybe, I don't know, but I did talk to Dr. Janelle Garcia at the IBM quantum computing lab and asked her about if that show was at all realistic, and she said it’s simulating nature, so a little bit. But I think that's a thing that is very far away. It's still in the sort of, ‘look at this cool research project.’
BP Right. Let's not discuss any products specifically, companies specifically, but another year in which I would say the dream of VR/AR kind of went nowhere, did not become more mainstream, have not found the killer app, and products that were introduced were met with at least as much skepticism as excitement, let's put it that way.
RD I think folks were trying the VR work productivity boosters, and maybe you can find a hologram room where it feels like you're in an office, but wearing a really heavy, uncomfortable hat all day so I can be bothered by my coworkers constantly doesn't sound great.
BP It's not the sales pitch we're looking for. So another thing that I wanted to talk about– well, I'll talk about the things that excited me this year and maybe then you could talk about the things that excite you. I'm very excited and interested to see to what degree trying to make AI more like humans will produce positive results and improve them and make them more useful, and to what degree we've passed that point. I watched a talk with some of the people who have been most influential in the field: Geoff Hinton, who's considered the godfather of AI, the head of DeepMind, the chief scientist of OpenAI. They had a conversation at MIT and a lot of them said, “We wouldn't be here without the brain. We were inspired by it. We bet on it. We took this architecture, but we may have gotten everything out of it that we can at this point. From here on out, we may optimize in a way that works better for computers and data, and that will be the path that takes us to the next level of intelligence or functionality or usability or value out of these things, not continuing to try to get closer and closer to what a biological brain is like.” So I'm very curious to see which way that goes. What we've seen basically for the last five years, or maybe even longer than that, is that some people keep saying, “All right, to get to human-level intelligence, to get to AGI or something, we're going to need to come up with new ideas.” And other people say, “No, I'm just going to do more data and more compute and get back to you.” And that just keeps working, so until that stops working, they're going to go that way. I had a conversation yesterday with a professor who, along with Andrew Ng, created the deep learning course at Stanford, and he said something to me. He was like, “As everybody says, by 2030 we'll be out of data. It'll have trained on all the high quality data. So at that point we might have to go a different way,” and I was just like, “Whoa, that's a weird thing to think about.”
RD Are people going to be paid to write novels that just go into large language models?
BP Right. If you allow it to jack in and view your dreams, you can make a nice side hustle with that. So I'm interested to see how far we get following the brain. I would say one thing that kind of pulls in that direction is that Google's latest big model– Gemini, is multimodal, and they taught it to understand text, audio, video and images in a way that kind of resembles our senses. They're anthropomorphizing it to a degree and it's to try to get it to understand the world and be useful, and the next step of that is embodying it in a robot like what Tesla has and getting that feedback as well. And so people are going to go in that direction. To what degree that will take us to something interesting remains to be seen. So that's what I've been interested in and excited about, that and the fact that computers can now actually read your mind, that that's not science fiction, that that's something they can actually do. At this time, it's mostly useful for medical patients who have a serious issue that requires a brain implant, but we are on the path towards some pretty freaky stuff.
RD Yeah, I can't imagine that one ending well.
BP I think my favorite thought about that one is what if you had to sign a prenup that said, “If you ever got divorced and you went to court, you had to wear this thing so they could just see right into your mind. You couldn't lie to them. They would only look directly at your thoughts.”
RD I think there's literally a Black Mirror episode about that very thing.
BP All right, Ryan. What are you excited about? What have you been thinking about? What gets you excited?
RD I'm excited about a couple of specific use cases. I'm excited about being able to take a PDF and ask it questions, and maybe at some point, somebody publishes a book and you can just ask the book questions. And those big ones that have hundreds of sources, maybe you can pull from those sources. I think that would be a great way to improve your learning, improve the information given by that book, so long as everything is accurate.
BP We're getting closer and closer to the imagined AI assistant in Her, where you say, “Read me my email. Summarize this. Go delete that.” People are homebrewing that right now; you can build that. It's not quite as capable as that AI, but it's getting closer and closer. And one of the things that you wrote– I don't know if it was this year or last year– that was most popular was the one about academic papers about computer science and the value of reading them and it would be cool to have something where every day you wake up and you've got this RSS feed and you're like, “Can you explain this paper to me in three minutes or less? What do I need to know, and what are the most interesting salient pieces of this? Explain it to me at a level that I can understand,” and then it tells you that. That seems like a cool thing.
RD Who else is talking about this? Who disagrees with this idea?
BP Yeah, I like that.
RD I would love to have a feed of translated international news that's like, “Here's what's happening in this particular neighborhood, according to very specific sources.” What's the city council in somewhere in Thailand? What are they arguing about?
BP Just because you're curious, or you're saying when you need to travel there, or as a way of viewing larger current events?
RD I think as a way of larger current events. I always loved an idea called the astronaut effect, and I think there was a site around it that just took YouTube videos that had default names that were very short and played them one after the other. And it was all just people's little videos of their lives. Here's somebody dancing at a wedding in Saudi Arabia. Here's somebody on a beach in Italy. Here's somebody's kids in South America. And it was over and over and you get this sort of warm glow of the aggregate of humanity.
BP The good stuff.
RD But I think one of the other things I'm excited about is people using the generative AI output to make weird things that you can't make otherwise. I forget who said it, but the defining feature of any medium is its flaws. You see people duplicating lens flares and fuzz and distortion from VHS tapes. So I think people using the weird creepy flaws that AI generated art has is sort of an interesting development for it.
BP I will say, to that point, Sony has a very prestigious photography contest that they run every year, and this year it was won by a German artist who then returned the prize and said, “This is not real. This is AI generated, and I made this as a provocateur to let you know that you may not be able to tell the difference anymore, and also that I think this is a new art form that should have its own category and that people can get good at in their own way.” So that was pretty interesting to see that.
RD And I think some of the constraints that people are able to put on generative AI like ControlNet, which I know has produced some of those where you can have text and it sort of embeds it into an image, or there's one with a spiral in a medieval village. Those are really cool.
BP The last thing that I was thinking about, which is a little bit challenging but also maybe exciting, is to what degree these AI are really going to disrupt the way the cognitive class works. The industrial revolution changed the way people do physical labor and large scale industry forever. As we build more and more intelligent digital systems, at what point are they really going to change the way “white collar” workers do their business? And the Stanford professor I chatted with yesterday said something I thought was interesting, which is that what will accelerate is the half-life of skills. You'll have to learn new skills more often, but you'll be able to do that in a more targeted way because you'll have this sort of second brain always with you, and I think that's especially relevant for developers. I think I read a paper once that said the software industry is, of all industries, one where you have to learn new skills most often because new programming languages become popular, new frameworks, you move from mainframes to internet to cloud, now AI, vector databases. You’ve got to learn this new stuff to keep up with the times and that can be quite challenging, but it's also quite exciting.
RD Indeed. We live in interesting times.
BP We live in interesting times, cursed lest.
[music plays]
BP All right, everybody. That is it for us. I am saying goodbye to the old year and saying hello to you in the new year. Hope you're having a good one. I have a great Lifeboat to kick off the new year, awarded yesterday to Corn3lius. An answer from Corn3lius got a score of 20 or more on a question that had a score of -3 or less. This question was saved from the dustbin of history. Corn3lius knows how to create spoiler text. Thank you, Corn3lius. No spoilers here, and you've helped over 65,000 people, of course. That's a lot of folks, so we appreciate you coming on the network, spreading a little knowledge, and congrats on your Lifeboat Badge. As always, I am Ben Popper. You can find me on X @BenPopper. You can shoot us an email with questions or suggestions for the show: podcast@stackoverflow.com. And if you like the show or you're feeling good because it's the new year, leave us a rating and a review. It helps.
RD I'm Ryan donovan. I edit the blog here at Stack Overflow. You can find it at stackoverflow.blog. And you can reach out to me on X @RThorDonovan.
BP All right, everybody. Thanks for listening. We will talk to you soon.
[outro music plays]