This week, we talk about code that writes code and how soon it will put us all out of a job. That, or it's a nice magic trick, and a great tool for accessibility. After that, Sara and Paul do some mock interviews, as we investigate a paper on the role of anxiety in the hiring process for software developers.
EPISODE NOTES
If you're wondering why GPT-3 matters and how it compares to prior efforts in this area, here is a good summary.
If you want to dive deeper into the effect anxiety has on the interview process and hiring in tech, you can read up on the research here.
This week's lifeboat badge goes to PerformanceDBA, who left an incredbily long and detailed answer, complete with charts and code snippets, on the following question: how to organize a relational data model for double entry accounting?
Transcript:
Paul Ford That's always the ultimate nerd word. When someone is just sort of like...vectors! I'm like, alright, you serious?
Ben Popper Yeah.
PF You know, here, take my money.
[INTRO MUSIC]
BP Satisfy the protection of your continuous integration and deployment workflows with Cloud Native application security, part of Trend Micro Cloud One. Get automated defense early in your pipeline and across cloud environments for visibility and protection. Discover more at trendmicro.com/stackoverflow.
Sara Chipps Hey everyone and welcome to the Stack Overflow podcast. I'm Sara Chipps and I'm here with my...
PF Co-hosts! [Ben laughs] Yeah!
SC Co-hosts...
BP She's searching for the right adjective there. Couldn't find something positive.
PF Enemies! My...enemies!
SC I couldn't think of the word co-hosts. I was like, they are hosts, but I am also a host. I need to be able to say that we are all hosts together. Ben Popper and Paul Ford. How's it going?
PF Oh, it's good. Everything's great.
BP It's going well!
SC Have y'all been seeing these tweets that have had stuff about GPT-3 in them? where people are like...
BP What's GPT-3?
SC So GPT-3, so GPT-3 is a platform from OpenAI is a language model and this is something they are slowly letting people have access to. It's an AI language model and it's been trained and lots of different ways, but one way that's really interesting. It is been trained to code both frontend and backend applications based on your requests. So that's been really interesting to see. So I've seen a few tweets that have been like, I need an app that will allow me to check the balance of my card and add deposit money. It'll quickly spin up a react app for you. It's not foolproof, but it does kind of make me wonder how long we're going to need people for these things.
02:08
BP Right, right, right. That's scary. The code writing the code. So when they trained it, did they just show it lots of existing apps and then, or like, how do you train that model?
PF I mean...okay. Is what I'm going to say. Oookay. This is very wild. When you see somebody type like make it yellow button and make it blue and you know, or make the text blue and have it say submit, and it shows up, you know, I seen a couple of demos of this online, but the limits show up really quickly with things like this because it's predictive, right. It's predictive based on a set of inputs. It's not as smart as a human being who has spent like an hour and a half reading through the widgets and dragging and dropping them where they want them to go. Right? Like the actual tooling we have for doing this work, it's very, very good. But I think that the, it is wild to see a computer do exactly what it is doing.
SC Yeah. It's it seems like something like this comes out every so often and people get hyped and then people are like, Oh, here are the limits of the thing. I wonder...
PF Well there'll be that tweet from the VC, which is like, ''guess it's time to get some new skills programmers!'' or like, you know, ''ever seen the future?! This is what the future looks like!'' in their, their Bitcoin dungeon where they tweet from.
SC I also have a Bitcoin dungeon.
PF I know. But there's is in like Morocco, like yours is...
SC Oh okay, not in Brooklyn.
BP The billion dollar off shore bunker. [yeah!] Well, yeah. I mean like the thing about this sort of demonstration is, yeah. Especially when you're using frontend web code, you can see how it's pretty easy to visualize something, you know, make a button that's yellow and then a donation thing that's here. The difference with that is like, you can't then say, and now securely connect this to my banking backend, like that's requires a human still, right? Like it can show you what it wants to do, but it can't actually integrate it into a system?
03:57
SC It looked like it was also setting up a backend, like a simple backend, I think it's, it can do simple things. I had a pretty disappointing weekend because my personal website is way, way out of date. Like I have like a press thing that hasn't been touched in years, like an area for press that, you know, has very old articles. And so I was like, Oh, this is a perfect opportunity because I'm just going to use this thing. And I'm going to say, cause I just want something with my name and a bunch of links because here's the problem, right? If I do that, I mean, it's not going to take me that long, but I have to send her a text on a webpage, which is really annoying. So it was like, this thing will be able to send her to the text. So I went online this weekend, cause I had seen all these videos and it turns out not everyone can get access. [ahhhh] You need to request. I was like, ruined. My dreams of redoing my website are ruined.
BP I mean, I do feel like even as a way of getting people excited about what code can do, if you can say like speak what you want the website to look like, and it appears before you, and then talk out your changes and see those changes happen. That's pretty awesome.
PF It's totally cool in gee-wiz, like the person typed a table of the richest countries in the world with the column names, you know, blah, blah, blah. And it's like, it kind of made it and I'm like, alright, that's cool. You couldn't do that before. You know, like two weeks ago, no one could do that. I don't know. I am very suspicious of giant AI models because the data kind of is a big blob. Whereas I love stuff like Wiki data, where if you learn how to query it, you actually have some sense that it's coming from an understood source and you can track stuff back. [right] I don't know. It, it is making interfaces in seconds that would still take minutes to build.
BP Right. [Ben laughs] Right. So like if you've gone from having to write the HTML code yourself to a drag and drop WordPress site to just saying what you want it to look like, and it's been set up.
SC But then you kind of have to fix it.
BP Then you kind of have to mess around and edit it. But the right, the reality is like, it's kind of like you teach it to play, Go. And after, you know, the first time it learns to play Go, and then in a year it's really good. And then in three years or three weeks, if you're training it on 3 million machines, it's like the best Go player in the world. So this thing...
06:06
PF Or, or literally it solves Go. Like it figures out every single common tutorial possibility.
BP So I could see this thing getting really good at sort of like dictation to web design. That seems like something it could learn to do really well.
SC Yeah. Kill CSS.
BP But then the reality is like, yeah, that might be nice when you're a small business owner who wants to, you know, like figure out and design your first website, it's not going to help you make a popular website or app. Like it doesn't have any spark of, Oh, people are really gonna want to use this. You know, like that still requires you to create something new, whatever the next TikTok is.
PF Here's the real application to me. Right? So it's not the free text field where you say, make me a banking app and it figures that out. Cause that, that is like, we all know that's a ways away, but what is this really it's like that when you're typing into Google and it tells you what the suggested searches are, because those are, those are coming out of a big model that's like this, right? So it's predictive. So what you could do, you are you using this to build software, what you could do is say I would like a rectangle or I would like a form and it would kind of know all the kinds of forms that are really typical. And so where this is really attractive is if your SDK or your widget library is enormous, which is, you know, a lot of big business systems or a lot of things that we do online. Like they're really kind of big, like, you know, social integration widget, or, you know, or certain kinds of, you know, modal boxes or whatever. [right] You could start typing that out and then it could say, Oh, well, you know, they're, they're saying I would like a, and I, you know, there's like 40,000 things that could follow. So let's, let's give them the, the most common. And then they, they, you know, you type the next letter and it sort of narrows it down. And so what it can do is give you access to a really, really big catalog of objects and then give you tools to customize those.
BP Yeah. I wonder also from an accessibility standpoint, this seems pretty amazing. Like if you had trouble typing or using a mouse and you wanted to just like jump into some web design, you could ask it to do things. You could look at the code and review it and maybe like interacting with all of that with your voice. That'd be pretty cool.
PF That's right. That is the good stuff right there.
08:04
SC It seems like AI has reached the level of college sophomore [Ben laughs] in their first internship. [yes] Where like, can do kind of lots of things, but not a lot of advanced stuff.
PF Oh, I'm trying to imagine what you would call that. It's like, it's, it's like the touring test. I don't even know what the term would be.
BP Yeah. But the thing is Sara, that, that's the thing it's reached that level, but it's only been out in the public for a little bit. So when they have a million people mucking about with this every week and they're collecting all that data, it just gets better, so much faster than a person ever could because it's like learning at scale. [yeah] That's the thing I don't know. So we'll have to wait and see.
PF Well, it can know all the things, right? This is something humans can't. This can have billions and billions of data points. It can see every react element in every react base by spidering all these different apps. Or it can look on GitHub and look at all the code. And then when you type something, you can say, wow, it really does seem like when they type this, the next thing they type is this. So let's, let's give them a nice interface to pick one of these things.
BP Right. I think I told you guys about this, but my son played the AI Dungeon, which I think was made on, on GPT-2. And right, that learned, you know, from the like very early text based role playing games, you know, you're in a cave, what do you want to do? And then you can say, and then it says it back. And I mean, he could interact with that for an hour and just feel like he was on the adventure. So that was pretty mind blowing. It took him down a weird path that that was a little unsettling at the end. And so I had to turn it off, but.
PF I just, like, I think for me, I just look at it. I've been looking at stuff like this for so long. So I sort of go like, Oh, well, you know, there's the magic. Alright well, I know how the magic works. Okay. Look, I'm just like, Oh, that's going to give you a lot of good menu options is really how I see it. [Ben laughs]
SC However it's a different VC every time that gets amped. It's like the new generation.
PF Oh yeah.
09:54
BP You're seeing that the slight of hand magic trick for the first time yourself. Right? So like everybody's gets the first time. It's really exciting. And then you peek under the hood a little. I mean, that's like when people say, Oh my God, it beat us in Go, it's going to destroy the world. It's like, yeah, it beat us in Go. It can't that same AI can't play checkers. Like it only does one thing well. It doesn't, it's not expansive. It's not adaptive, but I think we should shout out to anybody who's working on GPT-3 or has done some fun stuff with this. We are chatting about it. And we're asked to shout about it on Twitter, which means developers want to hear. So if you've done something fun with this and you feel like you have some interesting thoughts on its limitations or as possibilities, holler at us and maybe come on the show.
SC Yeah. We're dumb. Our limit of education is what we see with the tweets. [Ben laughs]
BP If you have better insights to bring on this topic, then we do...
PF Come talk about vectors.
[MUSIC]
BP Alright. So let's switch over and talk a little bit about this story that Sara shared from NC State News about tech interviews.
PF Alright, Sara, I read this article too. Let's do, let's do a little role play here. I'm going to, um, here's what I want to do. I want to interview you and I want you to be like a really together programmer who is really good at dealing with stress. And then we'll, then we'll switch. I'll be the interviewer. Hey Sara! This is the part where we like to do a little whiteboard test.
SC Great man. I'm hyped! What's up, man? I've been doing this forever.
PF Let me give you a, what's your favorite whiteboard marker color?
SC Oh man. I got to go with purple.
PF Awesome!! Alright, here we go. Now the purple one for the purple group. Let's get on that whiteboard. Yeah.
SC I knew you guys liked purple! You look like a purple kind of guy.
PF Yeah. Can you give me a little a, what, what do you call it? They're like a quick sort algorithm. Maybe do that in a, I dunno, assembly C uh, you know, or one of my faves, like, you know, I don't know, not Python.
SC Oh man, I'll do C. I could do with both hands behind my back.
PF Oh my God. Whoa. Purple crew. With the marker in the middle! Oh! Dudes! Dudes get in here, check this out!
SC Watch me do this. I've done this. I do this every day. This is, this is what I do.
11:54
PF [Paul chanting] Sara! Sara! Sara! Sara! Awesome! You have got the job, my bro! Alright. So that, that's the like, boy, that person crushed it. They did a really good job. [Ben laughs] They got up there. They put the purple marker in middle. Honestly. I didn't even see them do the algorithm. Just, it just worked out. Alright. Now let's switch. I'll be an interviewee, but maybe a little more on the nervous side.
SC Okay. Hey, Paul. Welcome. Glad you could be part of this interview.
PF Oof. Yeah. Hi. I was up late last night. I was doing the, I was doing a lot of interview prep.
SC Oh, wow. Okay. Alright. You're prepping for this interview?
PF Yeah. I mean, it's just like, it's a, it's a lot to hold in your head, but it is, um, I'm really excited about this role.
SC Oh, okay. Well, you know, hopefully, you know, this is going to be just your basic everyday stuff, so, uh...
PF Cool, cool.
SC Yeah. Hopefully you didn't do too much prep cause this is the kind of stuff, you know, you're going to be doing every day. So, uh, have you played the game checkers?
PF Uh, yeah. I mean when I was a kid. Sure. I did. I mean, this isn't real. This is like for a banking system. So I didn't think about that, but sure, sure.
SC Yeah. Well, you know, we get, we run into different problems every day, so here's what I'm going to need you to do. I want you to, so I don't know if you have anyone in your family that you have like a grudge against, but I want you to build a checkerboard and simulate a checkers game with someone that you maybe have some emotional issues with. And I want you to let them be the stronger chess player for most of the game and then beat them in the last 10 minutes. And then we have about 20 minutes for you to set this up and we're are asking you to do this at assembly.
PF Wait, you said chess and checkers, and then you said, okay, okay, hold on, hold on, hold on a minute.
SC Well you can do chess, if you want to. That's fine. You know, yesterday I had someone do an entire chess board and the pieces were actually looked like their family members. So I thought that was neat. That's why I put that in there.
PF Okay. Okay. So wait, so first of all, a board has it's eight by eight, right?
SC I mean, yeah. Yes. Yup. Yup. Yup.
PF And I'm playing, somebody has a grudge, but I want to win?
14:00
SC So you want to win, but you know, you're a little emotionally torn through the middle of it. So, you know, you, you definitely want to let, like, you want that to show kind of in how the game is played. I mean, you know, duh, you know, like, yeah.
PF This is for a backend API job. So maybe do you want me to like model it out as an API or because this is more like front end?
SC I mean, I'm not going to do this for you. So like, if you could, you know, make your choices, do whatever, do whatever you think is right.
BP Alright, alright. This is too much, we have to stop now. I've reached my limit of anxiety.
PF It's really that bad though. It can be that bad. It says, this is why like I'm a big fan of the paid take home test for this reason. Like, cause that, that is the real work. Like go, you're going to go figure it out.
BP Sara was this for a banking app? This sounded like more like an online gambling system where I would have to go lure them in and get the rake at the end.
SC You'd be surprised.
PF This is the problem. It doesn't matter. Right. It'll just be like, here's the arbitrary thing we just want to see. And then meanwhile, like dude A, it's just got that purple marker.
SC It'll be, the interview will be like, do a bubble sort. And you're like, but when's the last time you did a bubble sort? And it's like, well, it could happen any time. It's like, this is a front end role. What is happening? You know?
PF Yeah, no, I, that to me has never made sense about the industry is the desire to turn it into little and I get it because essentially it's just like, I can give them an SAT style test, you know, and they can show that they understand basic concepts, but that's no longer what it's like. Like no one Quicksort means you install the Quicksort library. If it's not already built into the language, it's just... A particularly motivated person or a person who's doing low level work. Like obviously they should be able to talk about those algorithms, but for a lot of API development and in particular, a lot of front end work, it's much more like, can you interpret the flame graph and talk about things that are going wrong in the JavaScript? So, so talk a little bit about this study though.
BP Yeah, yeah. What did the study find? The article that inspired this roleplay, what did it have to say?
15:59
SC I thought it was really interesting because it talked a lot about how the real assessing in these interviews is around anxiety and what happens with pressure. And it was a study out of NC State, North Carolina State University, and Microsoft on these technical interviews. It really just measured folks anxiety. And a lot of people were failing these interviews, not because they were unqualified, but they were stressed in the interview or nervous. And it impacted their performance.
BP Yeah. I mean, it definitely feels to me, and I was saying this before, that in the world of software engineering, there are these real time tests that I never have experienced in any other situation. Like even in journalism, we never did it quite like that. And in marketing communications, we never did it quite like that. I mean, certainly people will try to ask you tough questions in an interview, but the like actual act of throwing puzzles at people.
PF No pop, pop quiz.
BP Yeah. Exactly.
PF It doesn't, it just doesn't make sense to me. I don't know. I just like, that's never been the kind of engineering I've been close to and it's just built into the culture. I think it's a little bit like thesis defense and you know, they, they want to bat you around a little bit, but it doesn't surprise me at all. Right. And so this is a tricky thing because you actually do want to perceive people come to interviews in their stress position. Like Sara, when I interviewed you, your reaction to stress is to be like, you know, get further in there and grab the purple marker. Let's go. You know, whereas when you interview me, my reaction to stress was to try to get clarity about the problem so that I could, I could do a good job. Right. And so like, and what's awful about these tests and where the risk comes in is that, that doesn't give you an accurate portrayal of the person's ability to solve programming challenges. And so like real ones, like actual challenges day to day. And so like, that's why, again, like I like the test because you go away, you get your head around the problem. It's still stress. Like that is still a form of stress. Even if it's a paid test because you need to perform well and do a good job and that's hard enough, right. And if a person can actually do that, you can probably work out the other kinds of stress pretty well. You can resolve them. It's different. Like if I'm hiring someone for a sales role, I definitely want them to have a kind of equanimity and to be really comfortable and positive in the room. Product management design, there's different interaction styles that you need. But with engineering, especially in a large team, people can go, right.
18:24
BP It's interesting because you might say it's a little bit unfair to put people into these anxiety producing situations or try to measure social intelligence, which is what the like the purple bro marker test does to, in terms of like how good of a, of a, of a employee is this person I'm in this role and sales on the other hand, especially if these people have to do a lot of cold calls, that's really what you need to measure for, right. Like, right. Can you take the abuse? Can you turn the situation around? Can you charm people who don't aren't weren't expecting a phone call from you today?
SC Yeah. I thought something that was interesting too, is that they also showed that performance was just by adding someone watching the performance was reduced by more than half of people working on problems in front of an interviewer. There's just like such a safety in working alone, being your own judge or even, you know, working with someone you trust. I think part of the fact that the interview is just someone that is a stranger and that makes it hard because you were very worried about, you know, what they think and are they better at me than me to this? Are they silently judging me this entire time? You know, that's a really stressful situation.
BP That sounds so stressful, to be in a room with somebody while you're trying to solve a problem on a board. And they're just sitting sort of staring like, oof, that gives me the willies.
PF It's all really tough. And I mean, look, people, I still remember in my twenties places that didn't hire me. Right? Like I remember going to the interview and not getting the job and I will, I don't remember their names, but I remember the companies for companies all folded under terrible circumstances. I was very lucky not to get those jobs. Nonetheless, I will be resentful for my life. And I'm very, I'm very conscious of that. I'm very conscious of like, as someone who, you know, essentially I have control of a culture and then you're telling people they can come in or not. And sometimes you're telling them they have to leave and it absolutely earns you an enemy for the rest of your life.
20:07
SC Yeah.
PF And so it's horrible. One of my goals actually is to make the interview process positive enough that when someone doesn't get the job, they feel that they have been treated with respect. And the hardest part though, is you can't really tell them why. You can't be, you have to just sort of be like, this was not a good fit. If they ask specifically, you can, you can go in on the details, but in general you really have to be like, it's not right at this time. Because if you open that door, it's, you're spending all your time managing their expectations and needs instead of running the company. So that's really like, there's all sorts of this. It's hard to be helpful on the interview interviewer side too. And it's just, this is the worst part.
SC Yeah. It's a tough one. I really like the option. Sometimes people, there are people that are really comfortable in that situation that like excel when a stranger is watching them and.
PF The truly good interview is a very dangerous entity.
SC Yeah. Yeah. I like to giving people the option of demonstrating to you that they are good coders and whatever way that works for them.
PF Yeah. So anyway, alright, good. So what should we do, Sara, if we're interviewing and we're aware of that anxiety, what do you think? How should we start to factor anxiety into our interview?
SC Well, I think I've seen good interviewers and bad interviewers. What I've seen about good interviewers is just like encouragement, right? Like the idea that it's a stranger and being aware that you're a stranger being aware that that might be uncomfortable and being aware that people are worried about what you think, just being really supportive. And, you know, if they run into something like a mistake, you know, being like, Oh yeah, man, I've totally been there. I totally get that. And you know, just being in a place where we're focusing on the comfort of the person who is, you know, blindly showing you their skills, I think is helpful. What about you? What do you think is beneficial in this situation?
22:01
PF I actually think that having the script is pretty good because it, it keeps you, it allows you to compare person to person over time and make sure that you're, you have a pretty good process in place. And the script is not like you're reading it, but like, what are the four or five things that you really need to know? I don't interview many engineers. I interviewed mostly product managers and designers because it's more about client facing work in my life, but the basics are all the same. The one thing that I've really worked on is making sure there's a few gotchas in any interview. Right. And there, where you're just kind of looking for real risk points. And what I've tried to do is make sure those are offered very gently and conversationally rather than kind of, as you shut down moments that where there's no escape. Right. So it's sort of like, Oh, okay, well then what happened? What happened? You know, a big one for me is what artifacts do you produce when you're working? Like, and if somebody is like, wow, I don't really think about that. Or if they really go into word salad, that worries me because like, I need a, I need a written culture wherever I am. Like, I need people who are making things and can describe it like, well, you know, we do the pull requests or I do these big product requirements, documents. Those tend to be in Google docs, but actually there's a lot of diagrams that help people along. And I ended up presenting a lot of it as a deck. Like once I hear people talking about actually making the things in how they talk about them, I chill out a lot. And if I don't get that answer though, I don't want to set them up for like 15 more minutes of total despair. I'll just sort of, we'll go on to the next thing.
BP Right. I mean, I feel like I could see how this requires sort of extra effort or a better process on the part of the interviewer, which is to say, when you're interviewing somebody for a job, if they're confident and they're conversational and they can sort of respond to you in a way that makes sense, then you can start to assess maybe their skills or at least their, you know, their thoughts on certain problems or situations. If they get nervous, it's sometimes they go into a word salad just because, you know, like they're, they're feeling anxious and it's hard to actually assess how strong of a candidate they would be. And so you, as an interviewer, like you're saying, Paul would have to do to some, maybe extra work of getting them on a different path, distracting them and putting them back in a place where they feel safe to figure out if this person is a great candidate, but often if you have a lot of people applying for the same job and you want to get through 50 resumes, you just go with the people who you have a good conversation with. Cause you, you know, you're able to at least glean what that person is capable of.
24:17
PF Well, this is where you need a protocol. This is why we have HR, right? Like you do the, you do 20 minute phone screams to make sure that this person has like a good baseline communication pattern. And like, there are ways to make sure that time is being spent wisely. I have to say too, what's hard about all this is that the stakes are too high. Like it's high enough just to get a job. But now it's like, I'm also going to be giving you your health insurance. And you know, it's just sort of like, this is too much associated with a job. So it gets so, so serious. And everybody, you know, people are really pinning their hopes on that conversation. And so they're really, really anxious.
[MUSIC]
BP Should we hit up a lifeboat and call it a day?
PF Let's call it a day!
SC That sounds good.
BP Alright.
PF So this one has been now this one didn't hang out for too long, it hung out for three months and the question came with, uh, an entity diagram showing the database. I mean, it's good stuff, good stuff in this question. And the lifeboat answer is really long and explains double entry accounting, shows transactions.
SC Woooow.
PF Wow! This thing is colossal.
SC I feel like I have a business degree just after reading this.
PF I encourage, I mean, I'm going to sit and read this. It gets down to the SQL. That is really good.
SC Yeah! What a thorough answer.
PF So good job answered by performance DBA.
SC Performance DBA. Great.
PF And edited by how for so we should be really everyone. This is good stuff. Performance DBA asked the question. How for cure came in with that edit good work, everybody. Oh. Although antsy comes in ANTC and says, this answer is remarkable for two things. Apart from its length, it's a burn. There are no citations to establish practice slash textbooks. And the design of a business transaction is a single row is not used by any ERP or accounting package. So you know, what lot of voices in the room, you know...
26:10
BP There's no badges for backhand compliments and Stack Overflow y'all, sorry.
PF Oh, can you imagine? That'd be the whole...bam! What would it? It would just be like the back of a hand, like the opposite of the thumbs up emoji, just like the little hand...
BP Yeah. Somebody swinging a tennis racket back in. Alright. Well, thanks for listening. If you have thoughts on any of the topics discussed, you can hit us up at podcast@stackoverflow, or you can find us on social media. I'm Ben Popper, the director of content here at Stack Overflow and you can find me on Twitter @BenPopper.
SC Awesome. I'm Sarah Chipps. You can find me on GitHub @SaraJo
PF And I'm Paul Ford. You can find me at @ftrain on Twitter.
[OUTRO MUSIC]