Loading…

Building an agentic SDLC with a QA engineering mindset

Ryan welcomes Suneet Malhotra, Senior Manager of Test Engineering at Motorola Solutions, to chat about building end-to-end agentic SDLC pipelines using MCPs, using Cohen’s kappa to evaluate multiple LLMs-as-judges, and how you can improve requirements by shifting QA left through a specification enrichment stage immediately after the design phase.

The Stack Overflow Podcast cover art

You can learn more about Suneet’s five-agent SDLC (and see the companion code) on his GitHub.

Read the papers discussed: Cross-Layer Observability for LLM-Assisted Test Automation: A Reference Architecture and Web Feasibility Study and Cross-Layer Observability for LLM-Assisted Test Automation — Reference Implementation and Evaluation Data (v1.5.2, JSS In-Practice).

Connect with Suneet on LinkedIn and see more of his work on his website.

Congrats to user Sanjay for winning a Great Question badge for asking What does it mean to fork on GitHub?.

TRANSCRIPT

[intro music]

Ryan Donovan: Hello and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I am your host, Ryan Donovan, and today we are talking about what agentic SDLC pipeline actually looks like. And my guest has built a couple pipelines, agent harnesses to automate and create that from specs. My guest is Suneet Malhotra, who is a senior manager, test engineering at Motorola Solutions. So welcome to the show, Suneet.

Suneet Malhotra: Thank you, Ryan.

RD: Yeah. So before we get into agentic SDLC, tell us a little bit about how you got into software and technology.

SM: Yeah. I mean, in the industry for 20 years, did my master's in computer science from USC, originally from India, came here, started working in the QA automation in the media companies, Oprah Winfrey Network Ticketmaster. And eventually got, for the last seven, eight years, was working at bringing Amazon, Tinder, scaling QA efforts, automation efforts over there. So pretty much working backwards from the customer, making sure all the products we are launching for millions of users are pretty much high quality and making sure are meeting the expectations. And for the last six, eight months, working at Motorola, global team center as well.

RD: Yeah. So you've worked in a lot of engineering organizations. And, you know, as QA test, you get to see how it works or how it doesn't work in an engineering org, right? So you're looking at bringing the SDLC into the agentic era. What's the sort of things that you're thinking about changing, automating? What are the things you're building?

SM: Yeah, so automation in general has shifted a lot, like from starting from Selenium to Cypress to Playwright in the last few years for different automation. Now the focus has been like how to use LLMs, AIs, to help improving the velocity and doing automation, right? So the last two years has been like working with LLMs and just kind of figuring out how to obviously improve your velocity with the help of LLMs to write scripts. But what I noticed in the last six months starting, I would say December with agents and agentic sort of a coding. That came with the shift where now you give them the agency to do things on your behalf. And that's where I think the shift has happened, where you are able to leverage more with the help of agent encoding and really able to just prompt the AI, could be cloud code, codex, and then make sure you set the right goals, set context, and are able to help automate different use cases that you're looking for in the application.

RD: That is an interesting comparison with things like Playwright and Selenium, because those have a pretty deterministic approach with a pretty clear goal that you set for it. Like, here's how this test succeeds. But for agents, it's a lot looser, which is like, did this work?

SM: Yeah. So that's where last 20 years, more about finding specific object elements and make sure they are surfacing correctly on the websites or mobile apps. It's more about LLM with the visual regressions is able to look at the whole page and figure out, is it working as expected? So rather than focusing on all the element changes, locators, you are just giving the LLM the right sort of expectations goals. And especially with the last couple of weeks with loops, like everything coming in, you are able to kind of give the task and keep it running. And the goal is like, that's what the output you're looking for. And as a human, as a judge, you are able to come in and evaluate in terms of, is it working as expected? So yeah, that's the kind of focus has been leveraging that for mobile automation as well. And especially the challenge with different versions for Android, iOS, it kind of scales up like a lot of automation. It's very hard to maintain. So that's where with the help of these tools and browser stack, you are able to leverage it and really make sure you're able to scale up your velocity as well.

RD: You've built a couple open source reference implementations that show off how you think about these ideas, right?

SM: Yeah, yeah.

RD: So tell me about the five-agent pipeline. Where does it start? And where does it end? And obviously, what happens in the middle?

SM: So I would say each role, right? Let's say if you are a product manager or queue engineer or developer, everyone is started to work using agents to help improve their work. So if I'm a product manager, I'm able to leverage design tool MCP and really built a requirement document very easily now like with 70-80% accuracy. So what I noticed was like each rule is now independently working on sort of using cloud code codex and are using these tools to improve the velocity for a QA engineer just able to, hey, provide all the context about my requirements, Figmas, and able to generate the test cases, right, with the help of MCP. So each role has been doing it and leveraging that. And MCP is nothing but like a model context protocol. It just helps AI to talk to different applications, right? So just like what we used to do with API. So just give enough context for AI to be able to talk to like Confluence Jira. So what we did was instead of each individual role working across these different MCPs, why don't we try to launch something end to end? So what we did was starting with, you have a Figma design and as a product manager, I can have an agent which uses a Figma MCP and Confluence MCP, give the AI the context like, hey, this is the design we have, this is what I'm trying to launch, iOS, Android and all the kind of basic context. What I've learned is like as much as context I can provide to the AI that will help like improve my accuracy and really make all this process much smoother and faster. As a product manager you create a common document, right? And what I've noticed,

like it gets you to 70, 80% accuracy based on initial results. And then as a human in the loop,

you need to kind of still work with AI. Hey, you missed this negative test cases. You not really

thought about this, which are like a GTM strategy that I was trying to think. So you can work back and forth and you have a good sort of requirement document ready then that you can use that for second pipeline. So you have the requirements now. Then you come to developer, you work in a sprint, right? So instead of manually creating, let's say you are using Jira, all the Jira tickets manually, you give the context, this is my Jira epic, this is how we write it. Work with Jira MCP to create all these epics and tasks automatically, right? So that will help you again, save some time in that process. Now as a developer, I come in, I look at the Jira. At Cloud Code, 100% of the code is being written by AI, right? So that's where all the teams are leveraging it. So the dev comes in, they look at the requirement document, Figma and Jira, and they help start creating with the help of developer agent that we have created and really start executing those sort of PRs right away, right? So we went from product manager to a developer. Again, like if 20,000, 30,000 lines of codes have been written by AI, it's pretty much impossible for humans to check single every line of code, right? So that's where we kind of built in a PR reviewer agent, which focuses on making sure you either cloud code, codex, review each other's code and make sure you go through that process, right? And then comes in QA engineer agent, automation engineer agent, which goes through the same sort of requirements in dev code and make sure we are writing all the test cases with test rail MCP. And then we have our requirements being written. And for automation, we are using Appium with TypeScript and we are able to again use cloud code to just write all the automation scripts based on that, right? So we went from Figma to PRD, to dev agent, to a PR reviewer agent, to a QA writing test case agent, to an automation agent. And then eventually whole flow kind of put that into a reporting where you can use Slack or some other way to report the output from this process. So what we kind of, I notice in the work is I'm able to go end to end for any feature launch with the help of MCP agents and not really just working in silo with the help of AIs. And then obviously human in the loop is still needed on every single step. And that's the challenge. What I can notice in my work is like you get to a 70, 80% accuracy and how do you continue to work with humans reviewing the work on each step and making sure the end product is what is expected.

RD: It seems like with a sort of end-to-end automation like that, it'd be pretty easy just to let that run, right? And it sounds like you have built in a couple steps there, like building out the Jira tickets, the requirements review that sort of require a human in the loop. But what other steps are you putting in there to make sure that there's somebody getting in there and managing it, reviewing these sort of things that the agents are doing?

SM: Yeah. So like I said, right, when I noticed my initial, like working on this for six months, it gets you to 70, 80% accuracy. So first we noticed was we kind of standardized the agents with clear sort of a context about what your work is, what platform do you want to launch it in, what sort of requirements I want as a automation manager agent, right? So that's really helped me

to, first of all, have that harness, have that context to really make sure it's working as per my

expectations. It went from 70% to kind of 80%. And as we kind of continue to work every sort of

sprint or every two weeks, it really improves. So that's what I noticed, like it continued to improve and really continues to work on getting you 80% to 90% accuracy. One challenge I would still kind of notice was as a human, you are still involved, you can review the work. But let's say I gave a goal like, hey, get my automation to a 95% stability. So in the night just leave the goal, provide the context, what I'm trying to do. What I noticed like come in the morning agent commented out 20 test cases like hey, I met your goal, I was able to comment out these number of test cases and yeah, I accomplished my goal. Then you have to kind of have to work with the AI like, hey, can you make sure like you don't just comment out any? So I have to provide that specific instructions and make sure it doesn't do that again, right? So this is where the challenge I noticed like you and I started working on open source on a couple of projects. Like how do I evaluate the work that AI is doing? Because in this world, as we are depending on more and more on AI, it's getting very challenging for humans to really evaluate the work. So how do we leverage AI through that as well?

RD: Right. And when you reached out to me about making effective LLM as a judge, because that's sort of the paradigm that came up to alleviate some of the pressure on humans for a lot of this review work. So how do you get those good production evals?

SM: Yeah. So like I said, I noticed this problem, started doing some open source research. I could not find any good solution to what I was looking for. So I started working on it. So one way could be, so let's say you have a task, you have 100 or 200 automation test cases, visual checks you want to do. One way could be, okay, you get the AI to do the work. As a human, you review every single use case, make it pass or fail. Then you compare the work with AI. And then you kind of, after the grading of each work, you evaluate if AI is doing the good work or not, right? But again, that is very human, sort of a labor-intensive process. It takes a lot of time. It's not really scalable if you really want to try that across multiple projects or multiple features, right? So that's where I kind of leveraged a couple of AI like Claude and Codex. And what I did was had them evaluate as a judge, like a grader, in terms of evaluate the work of the third work that some other AI did. And what I did was did a couple of open source projects and had them do the work, judge every single output, and then grade it on a deterministic probability like zero to one, like what's the output overall, and give it a pass or fail. But again, the challenge in that is two graders working independently can still agree with each other like 50% of the time. And just by randomness, right? So let's say if they fail every single use case and 50% is my pass rate, I'll still get a 50% output, right? Which will be a false sort of a positive for me that, hey, I will still agree and depend on that 50% of the work, which is still not accurate, right? So that's where what I got to understand more is we can use Cohen's Kappa, which is a way to measure the work of basically the measure like, agreement between two raters. Which are classifying the same items into same categories in terms of like pass or fail.

RD: If they're agreeing a lot, they're going to have a higher Kappa.

SM: Exactly. So what we kind of did was like 0.7 is the where like, hey, this is the ideal sort of my confidence level threshold. Obviously, if they are agreeing, let's say around 0.8 to almost one between those values, this is amazing. Like I can really trust their work. Anything above 0.7 is where the threshold is. That's a substantial agreement. That's not just randomness that they are agreeing with each other. And that's where I can more confidently approve. And the remaining 0.3, whatever the failure that I'm getting, I can, as a human, I can still evaluate that. But that frees my time to just not focus on every single use case and just blindly trust the AI rather than focusing my sort of feedback or evaluation on the remaining 20, 30% of the work, the actual failures. And so that measurements, and I tried that on a couple of open source projects and your viewers can just download it and try it on some other projects as well. But that really helped me kind of define my sort of a threshold and accuracy level with that measurement criteria and really helped me in that process.

RD: That's interesting, the sort of agreement, like testing how flaky your LLM tests are. I wonder how much experimentation you did between different models having two or more models evaluating. Did you find that you needed the best frontier models to eval or could you get some cheaper model?

SM: Yeah, I tried Ulama, DeepSeek as well. Tried with those as well. What I noticed is a challenge. I mean, first of all, I think the ideal approach in this new world is like, have a harness built where you can use open router or whatever tool and really use different elements for different work. So let's say you can use frontier models for 10% of the rework that really needs the best sort of a model for the use case. And I can leverage some open source models for remaining 70-80% of the work, right? So I kind of built a harness as well where you can just not depending on one or two models, but try like four or five different open source models and frontier models focusing on the remaining 10% of the high quality work needed. So in my open source, yeah, I've tried more than a few models as well. And that's where I think that harness comes in. Like how do you build that sort of a code that is able to move from different models for the use case?

RD: In your pipeline, where's that 10% that needs the high-quality models?

SM: I think especially for evaluating the failures, that 20%, 30% of the work, that's where the frontier models comes in and kind of really deep dive on, like, what's the root causes on the failures? And that's where I think the most value I'm getting, evaluating that remaining 20%, 30% of the work. Overall, if you try a couple of model models, they will be pretty good at doing the basic work of just evaluating on the initial sort of a use case, test case that I'm providing. But the last 10, 20% is the work where I think these models really bring the value.

RD: Yeah. So I know some models have issues with sycophanty, where they sort of approve and they're being nice to you, basically. Did you find that any of the models were just approving, saying this test is good all the time, just like returning assert true all the time?

SM: It is, yeah, yeah. If you set up a goal, the model will try to best to like hey to meet the goal. And it could be either commenting it out or it could be say like hey, these are not hard failures so we can just not still pass the test, right? So I think that's where kind of having those empty files having that sort of a skills where exactly kind of you're providing exactly what you need from that. And that comes in like working day to day and kind of giving the feedback in terms of what you're looking from that AI. That really gets you to exactly what you are trying to get. So apart from just providing enough context, having those clear agents in terms of what you are expecting from that. I think the one more point of is like where the frontier models comes in and the planning part, when you're planning and try to give the inputs to the AI. That's where the planning is very important, especially for me using Cloud Code. It really helps, too, in terms of making sure what exactly my goal is to get from that task.

RD: You talk a lot about the iterative feedback loops. When you first were doing this, were there spots where the iterative loops exposed flaws in your assumptions?

SM: Definitely, yeah. One challenge, again, I noticed was in terms of no matter how much context you provide, it doesn't really take care of every single, like, incomplete design, right? So let's say we have a login test it will pass through basic, uh, sort of use cases for the app but it won't really have still enough context or take care of all the different scenarios like security, localization, it will still miss some use cases depending on your application, right? So no matter how much context or prompt you work on, but I noticed LLMs will still miss some of these use cases. And that's where I worked on the specific enrichment, again, one more open source. That's kind of complemented through the work that I did for LLM as a judge.

RD: The specifications requirements seem like that's where a lot of the weight is being put in the software development world. But in our blog, we've published a couple pieces that are like the requirements, specifications, those are the hard problems, right? They keep changing. You don't know what you want at the beginning. Do you have ways to sort of address those problems in the harness?

SM: That's still work. It's still ongoing. So what I’ve kind of been exploring is– you in the right, in the initial step, right, no matter how good prompt you built, you have to have understanding of like, what other use case that l’m is not thinking out from the beginning itself. So what I did was I put like this 16, 17 parameters that an agent will be checking before moving from Figma design to requirements and that will include localization as a pre-check accessibility and checking for like, security use cases, so I kind of build that kind of 14-15 steps right in the beginning of that phase from going from a design to requirement itself. So what agent will do is get the initial design, start working on the requirements, apply all these use cases for, like I mentioned, for localization accessibility, put that into the requirements. And then instead of just putting the document, it will ask the question to a human in the loop as a type of Q&A, like, hey, I noticed, like, do you need these four or five things? Like, do you need the localization for any country? So that really helped to kind of have that sort of a feedback loop with a human, could be a product manager or myself working on my startup. That helps, like, to enrich, like, provide enough sort of use cases which were missing in the initial phase. And that kind of has a cascading effect where if I have enough requirements clear, then it moves from a dev to a QA to automation. Every kind of phase already has enough sort of use cases implemented in that.

RD: Yeah, that sounds like that's the QA brain, right? Like you're trying to figure out where this breaks? And you're applying that to the specifications now.

SM: Exactly. And I can still apply, again, Cohen’s Kappa in the work case, like I can have two AIs do this work. And again, use the LLM judge to review the work as well. So still, I've tried on a couple of open source projects. It's showing me, instead of coming from 50% requirements to like 70%, 80%. I think I still need to do more work in terms of finding the right meta prompts, right sort of mechanisms to continue to try to find other use cases, which I might not have mentioned in my these sort of parameters. I think it's ongoing work. I'm still, still trying to figure out like where does that set the process of enrichment and what other metrics I need to provide to improve it further.

RD: And I've talked to some folks who in doing a sort of spec-driven development, use the spec not just to build the product, but also to test it at the end. Are you thinking about that? Or is that sort of like the Obama putting the medal on Obama meme, right?

SM: I think from my perspective, like it can run like bi-directionally, like from the initial phase to end the testing phase, it can run it. But from my use case, what I've noticed was if the requirements initially are pretty good coverage, solid, like in terms of enough requirements, then a QA engineer or a test user is able to build the test cases based on requirements. But yeah, this can work bi-directionally. Then you can take the output from a QA and then work backwards from it, like, hey, is it meeting the bar of the requirements? So you can work from downstream to upstream. But I haven't really evaluated that use case. I'm going to focus on enough requirements, then I'm making sure I'm trusting that sort of use cases and then building my test cases based on that.

RD: One of the open source is the specification enrichment. Is that where it's just you're working backwards to make a better spec from every point?

SM: Exactly. So basically, it's like a one extra LLM stage at the beginning of the pipeline. So like before any PRD or any test case is generated, it will enumerate like all the implicit constraints in the design so that it doesn't assume like design is pretty much correct from the beginning, right? It will do a confidence score like in terms of based on what it sees, where is the gap in the requirements and all the 15, 16 steps, it will provide sort of a, let's say it passes 70% of the basic checks from localization or what have you, it will pass that. And remaining 20, 30%, it feels like it's not confident, it's not sure if it's need to be added or not. It will provide a Q&A sort of a message for human, which will be me or any other person, human in the loop, and make sure they are able to review it manually in terms of based on the confidence threshold that they are seeing. And that Q&A I'll provide the input and then it will run that sort of a spec again and then provide the requirement document. So basically just a one more LLM extra stage at the right of the beginning of the pipeline. So instead of just assuming that design is pretty much 100% accurate or thorough, it just provides that sort of a constraint check which you cannot really build with a meta prompt you really need that checks and then QA with the human in the loop to really make sure it covers like, broadens all the sort of use cases that you are expecting in the requirements which a product manager might not have incorporated that in the initial stage, because like one more thing to add, right, in my 20 years, like, you come up with a basic requirement and then as a QA let's say you're launching in globally, right? You are supporting that in different countries, different languages. No one really thinks about these use cases in the beginning no matter as much as someone can think about it. And as a QA you are kind of then what happens is the requirements are written, and as a QA you come in the last stages of when the dev is already working on our product, you come up with the questions and the requirements changed, and then you are kind of working back in the last sort of stages of bug fixing or new requirements, right? Especially with the age of AI, like more and more people are using AI to write those requirements. That's why I think this step is very, very useful and needed.

RD: Yeah, it sounds like addressing a sort of historical issue with like getting the Q&A part to be further and further to the left in the cycle, right?

SM: Yeah, I mean, it used to be manually like working at Tinder. We would have this sort of a manual process like, hey, is it doing accessibility? Is it launching in US only or is it global? Are we testing for localization or not, right? All these steps, we were doing it manually anyways, even previously. No matter how much requirements you try to make sure it covers everything. That's where I think, especially for the global B2C company.

RD: Yeah. We're talking about specs and context. And I wonder how you think of the difference between specs and context, if there is a difference.

SM: So context is, let's say I'm an automation engineer. I'm working on a product. I have a spec, right? But that's not really enough. That's a very very basic key. If i just provide the AI just the spec itself will just provide me an AI slop of bunch of code or whatever based on just the requirements, right? So I need to be very very specific like hey, I need to provide the context first of all this is what my application is about look at my GitHub repo, understand the code base, look at what iOS versions I'm supporting or is it iOS web only or android only, look at what sort of how we work in terms of writing our use cases because each QA or use case will have a, sort of a difference. What basically prerequisite– are we making sure we have a prerequisite steps to reproduce all this stuff, right? Unless we provide all this way of how we write a desk case, what's our code looks like? What I play, what's our GT1 strategy, what platform are we supporting? And either you built that into agent, but you still need that sort of enough context be available. And even then, I would say it gets to you a 70, 80% sort of accuracy. If we just provide a spec, it's just AI slop, which will get you like a 30% bunch of text, which you will be just manually reviewing if it doesn't have enough context about what you're trying to achieve, what sort of a platform do you have?

RD: You got this running, well, it's getting about 70, 80% of the way there. How are you thinking about bridging that other 30, 20-30 percent?

SM: Every week you work with AI and of context, it improves. So it kind of gives you a better, better output that you're looking for. But I think especially with LLM as a judge, it helps me to kind of judge the work that AI is doing. So I can focus on remaining 20, 30 percent of the work rather than just living every single hundred percent of the use cases. So it's a matter of challenging in terms of, that's where I think it's the first 80 work is done. The 10-20 will get there eventually as LLMs improve especially with the help of loops. I'm saying it's– I'm getting better results. It's a matter of way like every– just keep up with the new technology how things are changing and then keep up. And I think in a year or two it should get you to 90-95 percent but that's the trend I'm seeing. Because initially when I started six months ago, we were there at 30, 40%, right? So the things have really exponentially improved in the last six months. So as we continue to kind of find more ways to leverage better context, better prompts, better integration among different tools, I think eventually we'll get there. But that's where I think the 20% is still very crucial, I would say, a part where the humans come in and especially all the coordination that you need to get anything out. Human coordination, planning still needed. That's where I think most of the work can be spent on that work of better planning and focus on the 20% of the work. So you can focus on multiple features at the same time and really improve your productivity. This couple of open source projects still work in progress. If your viewers feel free to download those and DM me if they have any questions, but really looking for the feedback from open source community, like how do we improve the work of LLM as a judge? How do we get better specific enrichment process in the beginning of the pipelines? So happy to get some feedback. And how can I work on improving that?

RD: Yeah. If somebody wants to get started with these, what do they do? Pretty simple.

SM: Just download the code, get any access to any cloud code, anything to run it and really evaluate the work. But as simple as that, I have instruction in the readme file as well if

they have any questions.

[music]

RD: Well, it is that time of the show where we shout out somebody who came on the Stack Overflow, drops the knowledge, shares some curiosity and earn themselves a badge. So today we're shouting out a great question winner, somebody who dropped a question that was so good it earned 100 or more points. So congrats to Sanjay for asking, what does it mean to fork on GitHub? If you're curious about that, we'll have the answer for you in the show notes. I'm Ryan Donovan. I edit the blog, host the podcast here at Stack Overflow. If you have comments, topics to cover, if you want to just say hi, reach out to me. You can email me, podcast@stackoverflow.com. And if you want to reach out to me directly, you can find me on LinkedIn. I post daily blogs about my open source projects or just anything happening in AI.

SM: So you can follow me on suneetmalhotra.com. And if you want to reach out to me on LinkedIn, search for my name and happy to answer any questions.

RD: All right. Thanks for listening, everyone. And we'll talk to you next time.

[outro music]

Add to the discussion

Login with your stackoverflow.com account to take part in the discussion.