BrowserStack is a software testing platform where David heads up developer advocacy and the open source office.
Connect with David Burns on LinkedIn to check out his podcast, BrowserStack Talks.
Congrats to Populist badge winner brentvatne for winning the badge on their answer to Read app.json (or exp.json) programmatically.
TRANSCRIPT
[intro music]
Ryan Donovan: Hello, everyone, and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I'm your host, Ryan Donovan, and today we're talking about QA and testing and why professional skepticism, which is the QA and tester's core skill, it might be the most valuable skill for all engineers. And my guest for that today is David Burns, who is the chair of the W3C Browser Testing and Tools Working Group, co-editor of the WebDriver Specification, and he also heads up the developer advocacy and open source office at BrowserStack. So welcome to the show, David.
David Burns: Thanks for having me, Ryan. I'm really excited to be here.
RD: Before we get started, we'd like to get to know our guests. Tell us a little bit about how you got into software and technology.
DB: I grew up in South Africa. I studied computer science in South Africa, then moved to the UK. But I was more interested in how people thought about using software because when I was doing my computer science degree, I also did a bit of industrial psychology because I couldn't work out if I wanted to be a psychologist or go into computers, really. And I think kind of it set me up really nicely because I did process re-engineering, which is kind of another form of automation. And then I got into test automation, worked for a startup, kind of started up their QA department, moved to Mozilla. I was there for 10 years working on automation and tooling to make kind of engineers more productive. And then I've been at BrowserStack for the last six years, kind of working on open source stuff, kind of telling everyone how kind of I think about QA, because there's never a right answer, but there's a right direction that we should be looking at. And how I approach problems, trying to make sure that kind of the web and what we see of it every day is high quality and makes us better people.
RD: We've talked to a lot of people building agents, and I think one of the most interesting ones was somebody who was a QA tester. And they built a full SDLC stack, but it was all about the evals, right? That's the big part of agents. So QA and testing is becoming a significant part of the agent development lifecycle.
DB: And I think it's mostly because... QA people look at things slightly different to everyone else in that they kind of focus in on like, if I were to use this, how would I benefit from it and try apply bits of empathy that they've built up over the years from speaking to customers, things like that, and then going, oh, this doesn't feel right. Like this feels like it's loading slowly. This is working this way. Also, they're very good at kind of being able to break down systems and then go this, just, is junk, right? Like you know I wouldn't develop software this way, I might not need to be good at doing it, but I can you know you see enough architecture after a while where you're going, yeah, this is definitely not the right direction to be going in. Reviewing things in a different way kind of opens up new avenues because they're constantly challenging systems because like, the bugs are already there, they like, they surface them a lot better that way. And I think kind of that way becomes especially in doing evals, AI, anything like that, you really push on that quite hard.
RD: It seems like QA testers do a sort of prove-verify. It's like, you tell me that this works, and all right, I'm going to make sure. Which, as we found from some of our survey results here, people don't quite trust AI, or if they do, they only trust it if they can verify it.
DB: Definitely. There's also, QA people tend to just double-check everything that they do, right? From developing their software, developing their frameworks, things like that. And if people don't do it, we lead to other problems that creep up. Like an example today, I got kicked out of Claude code, my kind of token had expired, tried to sign back in, and OAuth redirection sent me to localhost. And I was like, okay, someone's not looked at this today, but it's been deployed very quickly. The QA thing doesn't need to be a person. It could just be you looking at it from a different light, right? Rather than just happy path, just trusting what's coming to you all the time. We need it more now than ever before, this kind of critical thinking or kind of going back to the days of test-driven development. I still do it that way, but now I do a lot with agents to kind of help me move a lot quicker.
RD: So in the world of agentic engineering, what does test-driven development look like?
DB: I think it looks exactly the way it used to be with how we develop software. So you kind of have your PRD, your product requirement document, or however you do it as a team. And you go like, these are the paths I need to hit, like APIs, whatever. What I do is I have a two commit process, right? So kind of I have an agent go, right, you need to write the tests. Once the tests are written based on what we need, the next commit needs to be to make that pass. And what I always tell it is that you can't change the test without human intervention. So I need to approve any changes to tests because you've made an assumption when you're writing that code that I've missed when I've been reviewing it. And I want to double check everything along the way. And you can build that out quite nicely into kind of what you need. But that way, you've got a good amount of tests. In the past, people didn't write a lot of tests. So kind of they're now trying to do a lot of testing with AI, but they go, oh, I'll do it afterwards. But you're not kind of trying to prove that you're getting to the point that you want. And tests always help when it comes to refactoring later, because they will be refactoring, because you'll look at it and go, it's not performance. Like AI focuses on being correct, not necessarily all the other things like security performance, whatnot.
RD: Right. I've heard some people talk about tests almost as documentation for functionality. And it sounds like this is a little bit of what you're talking about.
DB: Yeah. And I think it's something that's, like I say, it's been there for... Agile Manifesto and all of the extreme programming that was coming out in the mid-2000s was all about that type of thing, right? So you can move fast, but try not to break things. Not like how Facebook talk, but you move as fast as possible with the highest quality along the way.
RD: I've seen writing tests as one of the big use cases for AI. Do you think people should hand over their writing their tests to AI, or is that something that humans need to be involved in?
DB: I think with any prompt, you should always review what comes out of it because kind of hallucinations are always going to happen, right? The AI companies have told us like they can't get rid of it. It's just a thing that happens. And AI is also a people pleaser, right? Like that's the key thing. It's like whenever you challenge it, it's like, oh yeah, good catch. It's like, just do what else, I don't need all the boosting of my ego, right?
RD: Yeah, no gold stars man, I don't need those.
DB: Yeah, exactly. So the human in the loop that people talk about is really important but especially in the QA point of view because like it can be documentation, it could be something else where you're kind of just reviewing it and going is this doing what I intended? Because if you don't have a good PRD and you just give it a blind thing, the next one that it builds on is going to cause problems. So this is where I think the push for doing everything through AI is not so good because it can help with that making things faster. But then you still need to kind of prove that it's doing the thing that it was intended to do in the first place. And that's where like QA people are really good because like intention is not necessarily, can I click this button? It's like, what happens when I click this button? Does it make sense for me to be clicking this button in the first place? Right? And that's where kind of humans tend to kind of use their lived experience in different ways and then look at it and go, oh, that makes no sense. Agents, can you change this in this way? Because I was thinking this way and you've gone that way.
RD: Yeah, I think one of the big parts of, you know, agentic engineering, prompt engineering or whatever, is trying to codify and processize that intent, get that intent in a sort of formalized manner. Do you think a PRD is enough to convey intent to the agents?
DB: I don't think so, because it was never enough when you gave it to a human to write code. It's the same thing. If you give me a PRD and you give someone else a PRD, like in the past, you would get two different bits of output, right? Like it might be converging on the right place, but the output would be different. And I think it's the same if you like ask a Claude or a Codex or whatever, right? Do this, you're going to get different things because it's interpreting things based on its knowledge, right? It's training data and going, oh, I think this, I think that, and then moving to the next bit.
RD: That interpretation part is interesting. Do you expect or do you think it's a good thing for AI tests to be a little bit looser and less specific? You know, sort of like the LLM evals?
DB: Like everything in life, I think the answer is it depends.
RD: Right. That's how you spot a senior engineer.
DB: Because where it matters most, I think kind of it should be incredibly tight. And those are the systems that if they fell over, could bring down everything. And there, I think we need to be a lot tighter. But if it's like, is this element in this space? And what you meant is that I've put an icon banner or something and it's in the top left. That should totally be incredibly loose because you never know what people are doing when they're resizing browsers or things like that when it comes to CSS. But then I was trying to sign into Claude code and it redirected me to localhost. That should have been caught by a test of sorts before it went out. That should be super tight when it comes to it in that sense and then building out from there.
RD: The local host seems like it's one of those, like, it works on my machine mistakes.
DB: I was always under the impression that, like, moving to Docker, moving to AI was supposed to be getting rid of those things. Yet, like, I'm sure we've all seen the meme of, like, ah, I've coded three different things. Here are the URLs, and they're all, like, localhost, colon, different things. Is that type of thing happening? And those are supposed to be memes. But, like, when a meme becomes reality, it's, like, you've got to laugh or you'll cry.
RD: That is interesting because it's a lot of the vibe coding stuff is people who don't actually understand the domain, right? Is there ways to give somebody who doesn't know anything a sort of set of PRDs that'll keep them from destroying themselves with badly vibe coded software?
DB: I truthfully, I don't know the answer to that one. I wish I could be a bit more kind of precise. But a year ago, I did a talk around like vibe coding and vibe testing. And I used kind of three months worth of data beforehand about how the world is kind of breaking under kind of people doing things or people ask the AI to do something and then it's like by accident it'll delete its production database. We're a year on and people are still making those exact same problems, right? And it's like, you would have thought kind of AI would have gone, hey, I noticed this is production. Do you really want me to do it? But like, you know, people are just so used to now kind of having prompts come up and it's like, can I do this? Can I do this? That they kind of going blind to some of those and then just clicking it through. I think that's in itself is a major problem, right? Like I remember when I worked at Mozilla, we went to faster release cycles, but we did a prompt where it's like, hey, are you sure you wanted to update? And people would just click through because they wanted to get to the next thing. And then that in itself then turned out to be more of a security issue, right? And why people are moving to Chrome is like Chrome just automatically updates it, right? People didn't care about that. But now like when there are these weird and wonderful like bash scripts and things that it's building out and you like look and go, I don't know what it's doing. Guess, go for it. What's the worst that could happen? But then it causes other problems really.
RD: The worst that could happen that like deleting the database, for example, you give it the ability to do this, it's at some point, you know, statistically possible that it does this. Is there a role for Q&A in preventing that? Or is this sort of like, you just need to restrict your agents, like lock it down?
DB: I think there's a lot of things we used to do in the past that I think we need to double down on. So in the past, you weren't allowed root access to the database. You weren't allowed root access to machines or things like that. There would only be certain people. And when they did it, it had to be incredibly explicit tasks to be able to go do that for security reasons and protection reasons. And I think we need to do more of that. And I think a lot of that can be built out into how AI does orchestration through skills and agents and things like that. The thing I found incredibly weird is that there's no standard way just yet in the market for all of it. I would have expected it from Claude, Codex, from Copilot, whatever. It's like, here are these things we ship with everyone, right, to prevent those scenarios. But we don't have them. And it then kind of looks like it's an easy thing to make a mistake on. And then there's all the security aspects of it, right? Because you just go, going back to our PLD example, it's like, hey, I'd like you to do this. And it's like, oh, I could do that, but I need to go fetch this package. And it just goes and fetches it. There's no auditing, no processing, no things like that. A lot of big companies used to put a lot of effort in the past. And I kind of feel like supply chain attacks, they're not getting better. Like, you know, getting lower in number. They're kind of becoming a lot more sophisticated and people are just clicking through. Yeah, yeah, I'm sure it's okay to run this.
RD: Yeah, there are certain things, you know, you should prevent your agents from doing. Your human developers shouldn't be able to do that either. Like no human developers should be able to delete the production database.
DB: I don't know how we got back into the stage. I kind of feel we have regressed at least pre-Docker stage in the mistakes that we're doing, right? And so it's like, I don't know why we're making these mistakes, but we're repeating them. And it's costing companies loads of money, right? We see large companies nowadays that you go to their status page and every other day it's down for that reason, right? And that's not instilling confidence in the products and things like that. And confidence is where QA teams tend to kind of exude. It's like, I'm confident this is doing the right thing, right? We're kind of regressing because there's fewer QA folk in the industry nowadays, kind of people offloading it to AI going, that's good enough, which I think is a mistake.
RD: In terms of building the confidence QA teams, do you have any opinion on formal proof languages like Lean or TLA? Because I've heard of some folks adding those into agentic engineering steps.
DB: I think they're kind of useful. I haven't used them too much. The way I do a lot of stuff is still kind of building it, using kind of the older techniques, but trying to build it out. And I think it's kind of moving the goalposts in those situations, going, oh, it must be that thing that does it better, where I think kind of we've had all the right things to do in the past. AI helps us write code faster, but now we're producing PRD is kind of a little bit slower because you've got to research them. You've got to do that. And AI is not going to help. It is to a certain point, but not everything. And QA, it can do the automation side, but automation is only one facet of what QA is, right? Security is a form of QA. Performance is a form of QA. And those are not easy to set up and run and then interpret the data and things like that. And that becomes hard. And are we still doing that side of things? And so I don't think changing tools or things like that. helps that much when we already have the tools there. We just kind of either stopped using them because they were old. We all went through that phase of like every six months, there'd be a new front end JavaScript API. Like there's a lot of feeling like that at the moment when these tools come out.
RD: Yeah. With testing tools, it seems like, you know, looking at the new survey results and with testing tools, it doesn't seem like there's any tool there that people have a great feeling about, right? Like this percentage uses it and a much lower percentage wants to use it in six months. Is there a generalized frustration with the tools that exist?
DB: Very much so, right? Like my experience in Selenium really shows that. But a lot of the frustration is because the tool is doing something and they don't know how to interpret it. Like for me, a really good example is for years, people have said Selenium is flaky. And then they started moving to Cyprus and then Cyprus started becoming flaky. Then they started moving to Puppeteer. Puppeteer started becoming flaky. Then they moved to Playwrights and like the same thing is happening. The same commonality in all of this is kind of it's a web application that they're trying to test. And when it becomes sufficiently complex, it starts becoming harder and like flakiness creeps in and they just don't know what to do. Like I was speaking to Philip Crick on my podcast and we were talking about flakiness because like over the years I've done a talk about your flakiness is nothing to do with your test framework, as an example. And one of the key things is that like it's how we interpret things are going to work. Right. So we all work with asynchronous languages nowadays. But when we're writing test scripts, they're incredibly synchronous. I will log in. I will do this. I will do that. I will do the next thing. Right. And then that mismatch then creates other problems because you're like, oh, I've clicked this thing. Why is this thing not here instantly, right? Or when I'm expecting is because the underlying system like JavaScript or databases nowadays and everything in between, they have some form of asynchronicity where they will like, I will go do that task and I'll return as quickly as I can. And then like the flakiness creeps in because their interpretation of how quick something should be or slow or whatever is different. And then they go, I'll just rerun it. Hopefully the next time it reruns, it'll be quicker and better.
RD: Is an actual solution to flaky tests like. If you start getting flaky tests, can you call it good if it's flaky 5% of the time?
DB: I think if you have a sufficiently large test suite, you kind of start working in those deltas. You know that 5% is going to be flaky. If it grows, then it's bad, right? So when I was at Mozilla, there was a guy I worked with, really fantastic engineer. And he decided one day he was going to tackle the amount of flaky end tests because people were losing confidence in the test, right? And I've always said QA is around confidence in the quality of the code, right? And so he said, all right, I'm going to create a website that's going to show all the tests and how frequently they are being flaky. And so it became a project which he called War on Orange. Started building it out. And he would start switching off tests, but then suddenly the flakiness would move to the next test because there's certain rules around automation that a lot of people don't necessarily follow. So to me, a good automation test would have three things, right? A known start, it would do one thing really well and as succinctly as possible, and then return back to number one, right? So set up everything back to number one. And people would just do one, two, and then kind of leave this like dirty state, which then creates flakiness, all these other things, right? But then people are like, oh, this test suite is flaky. It's like we looked at the data and actually we managed to track it back to this thing, which was the cause of it. And so I think there is a value in it. If you have 10 tests, then probably you should be pure green. If you have a million tests, and I mean like in that million is everything within the testing pyramid from unit tests up to end-to-end tests. You will have a number. I think that number needs to be agreed on by kind of all of engineering and how you deal with it. And then you work on kind of the delta of how that number moves because it becomes either bad or good. I know Google used to do that when I had a lot of friends working at Google. We were trying to solve the same problem in different ways. And then we all agreed. It was like, yeah, just measure the delta.
RD: Is there a case to be made for, like you said, in a big network system, there's a lot of moving parts. Is there a case to be made to sort of just limiting the test to a very small piece of it, just mocking, stabbing, everything else?
DB: Yeah, definitely. And that goes back to the testing pyramid. Testers know it really well. I think we've got to a stage where people do loads of unit tests, assuming the dev team do that part. There's been cases in companies that I've spoken to when I browser stack where they don't do so much, and they push the tests higher and higher up the pyramid. So they either invert the pyramid, which then creates other problems, or they do what I like to call the testing Christmas tree. So it's got a like wide at the bottom, goes a little bit in, then they get to kind of components and tests like that. And then it shoots out wide again and then goes to the top for end to end tests. And that creates problems because kind of you've got all these things where if you can limit how much you have to do to kind of the thing that you care about to make sure that is correct, then awesome, right? So I don't mind it spreading out for like say component tests because you can kind of limit where it speaks to, how it speaks to kind of like Selenium, Cypress, Playwright, Puppeteer. They all have network mocking to block those things so you can get the results that you want rather than having a system. And then when it comes to that end-to-end test, pick one test. and do that one test well. And that should be risk-based. So kind of you should have like one really good accessibility test, one end-to-end tests for this workflow, one for that workflow, like so that it makes sense. And then you look at it and go, we've got enough coverage that we have confidence and then put some other quality gates into it. So like add observability kind of tooling into it where you monitor it, how people are using, and then you can use that data to then build better quality gates later.
RD: I think you mentioned automation a little earlier, obviously with agentic engineering producing a lot of code, running a lot of code. How are people dealing with automation? Are people automating more or are people just bad at it?
DB: People are automating a lot more from what I've seen and the data I've seen. It's both good and bad, right? So kind of when I was just talking about you should have that one good test, that should be risk-based where people are like, oh, right. I've got this workflow, but I can have these kind of weird and wonderful characters. Let me have a test for each thing. It's like, well, do you really need that? Could that be a test that you put somewhere else in the system? So it could be a unit test or kind of a component test where you're kind of doing it so you can move things faster. I think one of the problems that I'm seeing is like. At least when I started my career, CIs were becoming the norm, like ThoughtWorks were pushing them quite a lot and like made sense. And it was always said the CI should run for no more than 10 minutes, right? And then people started like adding Selenium tests and it's like, oh, my Selenium tests take a day to run. It's like, okay, that's not right. I've seen the same with any other UI framework, right? Especially like, oh, my framework takes three hours to run and I've got it as wide as possible, like horizontally scaling it, but it still takes a long time. And then my engineers like, sometimes come back to fix it because like they've moved on to the next thing. AI is supposed to be moving fast, trying not to break things. And so, yeah, I see a lot of that.
RD: I had a conversation recently on this podcast about when intelligence gets commodified, the thing that becomes more valuable is judgment and taste. And it sounds like you're talking a lot about QA judgment and taste, not just automation, not just automated tests. Do you believe that the QA testers sort of perspective, taste, judgment will become more valuable in an AI automated world?
DB: I definitely think so. And I think it'll be especially true for how senior testers do their jobs,
like automation or exploratory testers, because one feeds into the other. Right. And I think that taste becomes really important. So I was having a chat with Marat, who's really famous within the testing space. And she has this phrase where she's like, when you're doing some exploratory testing, you're gaining knowledge to then move into the automation space. And if you get that right, that gives you more freedom to go back to do that taste or things where you're looking at it kind of from a how a human would interact with it. Like, we still are going to be using websites, we're still going to be using mobile apps, I don't see that changing anytime soon. There might be kind of improvements in like pathways for AI into it, but I still like, you know, if I'm going on holiday I want to have a look at the photos of the hotel and things like that to make sure that I still want that and I wouldn't hand that off to AI. But I might go, hey, could you like clean up my email, do things like that, and it will do that really well. But those pathways will be very different. So I'd rather spend my time on kind of the things that are important and the things that are important to people and to their customers tends to be like, how is my interaction with the application I'm using? Like, is it fast, slow, secure, not secure, whatever, right? And go from there. The main thing for me is like when you're getting frustrated with testing and kind of flakiness, see what you're missing in your knowledge base for how that application works. I think that's the key thing. And then remember that a lot of AI, it was trained on information from three, four years ago and bases everything off that. So if there's new things or kind of new ideas coming up, review everything. AI wants to be your friend and wants to be a people pleaser, but a challenge everything that it does.
[music]
RD: All right, folks, it is that time of the show where we shout out somebody who came on to Stack Overflow and earned themselves a badge, dropped some knowledge or shared some curiosity. And today we're shouting out the winner of a populist badge. Somebody who dropped an answer that was so good it outscored the accepted answer. So congrats to Brent Vatney for answering readapp.json or exp.json programmatically. If you are curious about that, we have the answer for you in the show notes. I'm Ryan Donovan. I edit the blog, host the podcast here at Stack Overflow. If you have questions, concerns, topics to cover, if you want to reach out and say hi, you can email me at podcast@stackoverflow.com. And if you want to reach out to me directly, you can find me on LinkedIn.
DB: And I'm David. I head up developer advocacy and open source at BrowserStack, working on all your favorite testing frameworks. While I'm there, I have my own podcast, BrowserStack Talks, where it kind of talk about QA and things like that. So please do come follow me. And if you want to see what I'm writing about, I've got a newsletter up on LinkedIn, and you can find it there.
RD: Thanks for listening, everyone. And we'll talk to you next time.
[outro music]