This episode with PayPal’s CTO Srini Venkatesan was recorded at the Ai4 conference.
Listen to our other Ai4 conversation with Greg Jennings, VP of Engineering for AI Products at Anaconda, about building secure-by-default AI coding agents here.
Connect with Srini on LinkedIn.
TRANSCRIPT
[intro music]
Ryan Donovan: Hello and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I'm your host, Ryan Donovan, and today we are recording live from the AI4 conference. We're talking about, obviously, AI, but how you can keep it safe, trustworthy, not steal your money in a financial organization scale. And my guest for that is Srini Venkatesen, CTO of PayPal. So welcome to the show, Srini.
Srini Venkatesan: Nice to be here, Ryan.
RD: We just got off stage, the keynote, talking about similar things. So I want to sort of double tap on some of the things we talked about. Obviously, PayPal has massive, massive scale. 400, was it 300?
SV: 400 plus million customers.
RD: Me too, you got my money. But it's at scale and it's with money and you're introducing AI and some things. How do you think about the sort of safety versus adding benefits?
SV: So it's a great question. We are in a very tight financial space. We move trillions of dollars, as you know. So it's important for us that there are two aspects of AI that we worry about. One is how do we make sure it's secure and it's trustworthy. The second part is also ethical AI. And how do we use AI in a consistent way? There is no biases in what we do. So let me talk more about the first one, which is how we think about how do we keep it secure and safe and so forth. The key for us is validation more and more. As we go from code assist to loop engineering to automated autonomous SDLC, the key is how do you validate it? How do you make sure that the checks and balances are there? What, what we have done is we've focused more on the validation part of the equation even more so than the coding part of it to make sure that it is consistent and very clear. We still have human in the loop to make sure that the human is verifying everything that we do and we can get that right all the time.
RD: So you talk about validation and usually that means sort of LLM evaluation, right?
SV: It's both LLM validation and also actual functional validation. So the key for us is, so the challenge with the AI is it's more gen AI and LLMs are, it's more predictive. But what we do is like, we actually made them write deterministic code and then that is what we use to validate rather than like, let the reasoning validate themselves, because then you get inconsistent results. So what we try to do is like, yes we have the reasoning, now to create the deterministic piece. But where we are want to be deterministic we are trying to be deterministic. There's no point in predicting something which can be deterministically done. 90 percent of our validation is our deterministic validations. We could have generated that using LLMs, but they are not runtime reasoning decisions.
RD: Yeah, and that makes sense, especially when you're validating numbers. LLMs are not great at numbers. Something I've heard recently is using proving languages like Lean or TLA+. Right. Do you do any sort of validation through that?
SV: Not right now. The change is happening so fast. For me, it is about enterprise grade. How do we take every step, make it enterprise-grade, get it out there for general consumption
within my engineering population, and then go. We are leaning into stuff like foundational models and all that kind of stuff. But right now, the goal is to take it up a step-level function within the company and keep progressing that way. Because getting 100% of two engineers is not as much as getting 20% more productivity from every engineer. And so enterprise scale is important for us.
RD: And that enterprise scale is tough because I've heard of folks starting with a five-person prototype, a testbed, and then scaling that up. Did you start with a small team?
SV: So what we did is we started slightly differently. We basically initially said we provided all three tools, Cloud, ChatGPT, and Perplexity to pretty much every employee and every engineer so that they can get familiar with that. They chose to use their tool offering, whether it was
CloudCode, Tursor, CodeEdge, or Copilot, to do CodeAssist. So we get them to get familiarity with it. So every step was to get everybody familiar with it. And then we started doing loop engineering with two or three teams. They developed expert systems, and they created autonomous STLC. There are three or four teams which are foundational teams that we are getting autonomous sales TLC going, okay, now the next step is, how do I bring together this in a like a open standards framework within the company to kind of– because there will be some differences in the harnesses, but this principle is knowledge layer followed by an expert layer followed by a workflow layer and then a tool of your choice. So we are standardizing on those stats and that's kind of how we are doing it. And we'll hope as in the next three months, we will roll it out to the engineering organization from that perspective.
RD: Yeah. I'm curious how you're thinking about loop engineering. This is something I think it's been very prominent in the last six months or so. And so we talked about the Ralph loop. Yeah. Right. How do you create an engineer loops for safety, for iterative feedback?
SV: I think the key is there are three things that are very important. You need an SOP or an expert. You need a base understanding of what you want to do or what is the expert layer.
RD: You have a goal. You have to have a goal.
SV: You start with that. Then you want to make sure that all the tools that are needed to get to the outcome you need are exposed through an API and through MCP so that the brain can control the tools. And then you need strong evals. Evals is the most important piece of it. It's like, how do you make sure that you're evaluating it? Be it engineering automation, be it servicing automation, be it any kind of automation, you need to have strict evals that basically runs through it.
RD: You need to figure out how to evaluate whether the goal has been reached.
SV: Coding is slightly easier because you can write functional test cases and integration test cases, and that's your easier evals. So that's kind of how we are in engineering. We have done the PESI, which is the inner loop. And then now we are working on the outer loop, which is how do you release, experiment, monitor, and activate, right? So you have to get both the inner loop and the outer loop are done. For example, we are moving to cloud modernization. For us, from a monorepo, how do we take it all the way to the cloud where it can provision you your compute, it can provision your network, it can provision your database, all in a single clip is kind of where we are automating right now.
RD: Yeah, that's an interesting one because that's an internal money trust, right? Like, if you're
provisioning stuff in the cloud, like, that's going to cost you.
SV: Yes.
RD: What sort of trust controls do you have on that kind of provision?
SV: So, obviously, we have an established FinOps functionality. If you're on a cloud, and more recently now AI, you really have to have a great FinOps. The point is to do flexibility, but while being very efficient and effective about it. The principle that I usually adopt in organizations is, you're going to get access to anything so as to be able to get a concept up to life, because time to market is important. Then you optimize account of it platform to really get to it. Because the lifeblood is innovation. You'll never sacrifice innovation. But then you have a platform level diagnostics and monitoring and teams which are stripping out the efficiencies and the effectiveness.
RD: That's interesting because there's a conversation coming up about FinOps for token spend. And I think in a lot of conversations, people are like, that's something we should talk about. But everyone's like, no, no, no, let's get it right. Let's get the product. And then let's worry about our tokens, right?
SV: I think like it's going to be a blend. The total cost in my mind can be a runaway cost. It can run away very fast. So we are doing this in three phases right now. Yes, we still, let's get it right to go there. That's one part of it. But you can allocate a budget to it. But the other end, what I'm trying to implement is we will get usage controls. Everybody has a limit so that you don't get this extreme outliers to make sure that you're protecting against runaway costs. The hybrid approach, which I think will come in the next few months, is what I call authorized usage. What I mean by that is when I talk about autonomous SDLC and we have an harness and we have multiple agents, when we build the standard harnesses, the harnesses will have built-in ability to choose the right model for the right task. The point is, for the reasoning, when you're a lead and you have to figure out the architecture, Opus is the best model. But you don't need an Opus model to write code. But this seamless switching in and out of the model based on what needs to be done is an important piece. Yes, there are LLM optimizer skills and all that kind of stuff. Cloud code does it. But when you know the end-to-end flow, and you are now creating an harness which knows when to reason and when not to reason. That, I feel, is the effective thing. So that's the guidance I've given to my team is if you use the standardized harnesses, your total usage will be much more. If it is not one-off, one-shot, you'll be subject to cap limits because that's the only way to effectively control it.
RD: Right. Yeah. So you've built your own harness on top of LLens. You know, you talk about the inner loop and the outer loop. How much of those loops does the harness cover?
SV: So the inner loop pretty much covers everything, right? So there is a lead agent that waits up and says, okay, this is what I need to do, loop, and then fires up an architect agent. The architect agent goes up, looks at all the domain knowledge experts, and then actually creates the high-level design, right? Then there's a transfer from there to a low-level design, which is a senior engineer, which basically takes it to an LLD. And then there is the implementer, other coder, a developer agent that actually does the coding piece of it. Then there is the test agent, which basically adds the functional tests and the integration tests. And lastly we have the validation agent which can review it so that's the fact we cover the entire cycle on the inner loop on the outer loop, we have the same thing; release, build, release, deploy, right? Once it's thrown over the wall over the wall how do you do canary deployment how do you test it? So I think those it covers full spectrum of it, yeah, but what is important is, how do you describe the task in a jira or in a story right? The more comprehensive you are, all the agents would to redraft and do it very well. But the interesting thing that we found out was when we combined the inner loop and the outer loop, so it was interesting. Something interesting came up. The inner loop did their code and actually deployed it. And then the outer loop failed because the functional test did not pass. That feedback went to the inner loop. And the inner loop agent... so RNS actually now started to learn from it and started coding the functional tests. So now what we're seeing is that memory is compounding between different layers. So it's kind of like a group of engineers coming together and learning from each other. So that memory is going to become the most important thing. How do you self-reinforce? How do you keep learning? If you really look at an organization, it is a bunch of memories that are found by a lot of engineers over a period of time. And if we can slowly encapsulate that into the agents while preserving the true architecture and the expert layer, now we have the best of both worlds.
RD: I mean, as an employee of a company with a knowledge management system in Stack Internal, we understand that very much. The coding agents left to their own devices, do whatever is sort of statistically median, right?
SV: Yes.
RD: But, you know, there's a lot of things like what dependencies are clear?
SV: Yes.
RD: What coding practices are okay? You know, what's the process to get this live? Do you find that coding agents respect that when it's given as sort of…
SV: Yes.
RD: Yeah?
SV: Yes. So for the cloud modernization that we are doing… we have given a cloud, we call it cloud radar. That is our expert system. That's our documentation of what should be the patterns for cloud. The point is they respect them. But there are two ways we do it. Not only we do it because we have validation stuff in that too. It also determines validation. It will allow it not to get out of... the key is when it tries to create something... do you want it to create it every time on the fly? Or do you want to create a code that it invokes every time that it needs to run through? So what we have focused on is in the later. Because if it is in the former, you have to constantly keep watching out for these anomalies. So that's kind of what we are doing. So as long as it is codified and scriptified, LLM scriptifying it, it honors it very well. Now– few escapes I'm not saying it's perfect, but I feel like the accuracy is much more that of an engineer right now.
RD: Yeah hey, I mean like, I, I've found that people are talking about getting much more prescriptive about what the processes are. Yeah, you know I had a conversation yesterday that these agents are very goal-seeking, not always in the same way we want them to do, and you said there's, there's a couple escapes? Those are always like a little bit of learning, right? That's feedback.
SV: They're learning.
RD: Can you talk about some of those escapes and how you sort of locked them down?
SV: So, so, the, the escapes from example is like, about like, the code validity. When I said escapes right, what happens is right now like if, if you are not knowing what the upstream functionality is and the downstream functionality is you basically code exactly to the API spec and it will work very well. Right, okay, but the problem is the upstream system might be expecting it sending you a different data and that might then not cause the right results to happen on an end-to-end test basis. So when I say escape, the escape is the end-to-end, not working very well. So that's where what we have now done is keeping on expanding. So the goal is, entire PayPal subsystems, we can expertise it. Because the more we give it the fuller context, that it knows it's all the context. Then what happens is it becomes itself its own governance and limitation. So that's what we have found. When two escapes happened, it was because it did not know the upstream system and the downstream system. So now we have started like, so every team is now branching out into developing that expert layer, right? And then how well we document that expert layer is also important. It's a learning, but it's actually, it is incomplete information. The analogy I'll give is this. Even in the engineer in the past would have made a mistake with the code because they do not want the services. They deploy it to production and then like they say, oh, upstream did not expect it this way. It's the same problem you're seeing it. What is happening is it's so fast and so accelerated that you're kind of having to respond with it real time.
RD: It's interesting the sort of identified SDLC where it's like you got every piece there. So what did the human developers do?
SV: So the human developers, there are three things that we try to focus on human developers on. One is to make sure that the expert systems are pretty clean and updated. because constantly code is changing, which means they have to make sure that the expert knowledge layer is kept up to sync. So that's number one thing that they are doing right now. Number two, right now, remember I told you manual review is still there. So everything has been reviewed. So they are manually reviewing everything that needs to be done. The third is the... volume and the productivity is going up, like they are able to do more stories, they're able to do more PRs. Right, so that is compensating now. Velocity is starting to keep up, right? The challenge is still you need to get the full ecosystem running entirely to get the full productivity out, so that's kind of what we are focusing on now.
RD: Yeah, and, and making sure that you know, you might be shipping more PRs faster, yes, but is it, you know, good at the end? I have heard sort of anecdotally like some reports of code quality dropping but also people are like does that matter in an agentified world right?
SV: I, I would not say see for, for us being a financial company code quality matters very well very much to us so we have not sacrificed on that. Right, so that's what I would say our review process and our validation takes a little bit longer, so we haven't gotten all the savings that we can because I just want to have that extra layer of defense because we cannot sacrifice on that. Quality is paramount for us right, like we want to be operating in the six nines now for them. So yeah so it really matters to the U.S. I think what it really helps out is there are a lot of what I call PRs which are death threat. PRs the death by thousand paper cuts kind of PRs, which is, you ought to fix this vulnerability, you ought to fix this upgrade, this package, you ought to do this. All of those things, those are the things where we are seeing a lot more– it actually is improving their quality because now, now I can upgrade every open source patches out there. I can upgrade patches faster. I can upgrade the zero day software is faster, right? So in reality, it is helping that quality go up faster. On the other side, we are making sure that we have the review process so that we are validating it very well.
RD: Yeah, I thought that, you know, some of the best use cases for AI are the most boring. And I think, you know, in the talk, you talked about helping merchants upgrade. And I was like, oh, beautiful. I've seen people, you know, sit on upgrades for months, sometimes years. Because you've got to check the whole code at that point. How are you helping with the upgrades?
SV: So there are three friction points in that upgrade, I think. One is that's your own proprietary software, which is calling our proprietary software. That's number one. Number two is validating it. And number three is certifying it. So because we certified given the financial and regulatory concerns we have to certify it because at the end of the day we will not accept payments until the solution is certified, so that actually works to our advantage because we know how to certify and validate given that we can do certification and validation. So now what we did is, how do we remove the friction of actually changing your software, right? So after six months of using internally STLC processes and autonomous, we discovered why can't we take it to our merchants. So what we did is, similar to CloudBot, we basically built our own purpose tool called Maya, which is a small software that downloads onto your computer. It can use any LLM of your choice in your hosted environment, but then it is sitting in your secure environment. It is looking at your code, and it has the expertise of PayPal's knowledge system. So then it's now looking at the old integration and then saying what it will take to update to the new integration. So that is how we automated it. And that we put it into a loop engineering because we do not know, because we have millions of merchants and they all have customized integration. So now this reasoning and this patent management is now really helping us. So before we did rules-based stuff, right? It helps you manage one or two things. And every time you see a new... there's going to be some fantastic edge cases. Now the thing is… combining the deterministic state with the reasoning to see new patterns and adapting and self-learning and preserving the memory so that it learns from that. That is what has gotten this real uplift for us.
RD: Yeah. So I wonder, you know, we talked about like, you know, CTOs, CIOs would be loathe to let agents touch their production systems, especially on money. What kind of... testing, and how do you prove that to merchants that they can trust it?
SV: So at the end of the day, see, if you really look at even prior to AI, testing was either automated or you had some functional testing that you had to do. Right. So now, if you really look with Gen AI, browser testing has become much more simpler. It's now able to comprehend, test, and actually go through it. At the end of the day, testing is two parts. One is is the customer getting the outcome that they want. What I mean is the edge end consumer of the merchant, right, getting the checkout or pay for payment that they need to do and get what they want. That's number one number two is, it secure and no failures and no errors are happening, right? The second one is much straightforward. It's about looking at the lots and looking at errors and all that kind of stuff.
RD: You can spot a failure pretty quick.
SV: The first one is also becoming easier with Gen AI because now it's easy for me to actually go through a checkout of any site. Prior to that, you have to know to learn the DOM or the HTML and you have to figure out how to click it and you are creating custom use cases. Now with Gen AI, it is able to have camera vision, it's able to take the browser. It's able to figure out what is the checkout button. It's able to see what fields are there. So that part of automation has also now become much more generate. So now that is how we are able to validate it faster. So it's not just coding. It's also the validation has become much more simpler. So when you do those two, then your level of trust is pretty high.
RD: Yeah. And you talked, this is a big friction reliever. I think the original PayPal button was one of those big friction relievers because you didn't have to type in cards or anything, you know, you're giving people a additional sort of freedom on the internet. But there's always the trade-off, freedom, security, right? How do you balance that for the the sort of agentic things that you push towards customers?
SV: So if you really look at you talked about the PayPal button, right, the PayPal button was basically, it was two things that we did. We were certified that the merchant was reliable because we made sure that button was not put anywhere everywhere. Number two, we also did buyer protection and we certified it, yeah, but if you really look underneath that we were doing a lot more other things that we are constantly waiting to make sure transactions are legitimate and so forth. What we have now taken at one level above which is what we call kind of endless check out or think about it that way right? So there are two parts that happens in a checkout: you are filling in all the information to make sure that the payment transfer is legitimate, but at the same time, you're also creating a order into the merchant system actually do the fulfillment and logistics. So in the checkout form, you're filling this information, and then that information is sent to the payment information to make sure that it's a legitimate money movement, and then it goes up. But actually, there are two orders that are happening. So what we have now decided is, why don't we swap the order? We are anyway doing payments. We get all this information. We have the wallet. We know you, all of that. So why don't we pre-fill it? And then we drop the order otherwise. So now what happens is the checkout form completely goes away. So when the checkout form goes away, it's a one-click purchase, which is what we talked about in IA4. That one-click purchase, now basically what it does is it removes the validation friction, information sharing friction, and the financial instrument sharing friction, and it basically is a seamless purchase. The beauty is because it's headless. Not only can you do it on the internet, you can view online sites. It could be on the social commerce site, like a feed with a one-click button. It could be on a television with a linear…
RD: To scan a QR code.
SV: To scan a QR code or use a remote to click a button. Or it could be in the future Agentech as the Agentech ramps up. So that is what we call is the future at PayPal. And that is how we are taking the PayPal into a headless... all the trust, identity, infrastructure given to you in a single click.
RD: So I can buy dumb stuff on the internet faster, right?
SV: Faster, yes.
RD: Yeah, I mean, it seems like there's a lot of things going headless in the age of AI, where it's like your back end is connected to whatever agentic front end there is. Do you think that is a good thing, or is there something lost when we don't know who's behind it?
SV: Imagine in this world right now, right? We have user interfaces, but what has happened even before AI came or Gen AI came, I'll be very specific. We did do personalization. Okay. So personalization was nothing but some models telling you for Ryan, I need to show these things. Right. Okay. So those were all design time bindings. Our models were pre-developed. Yeah. What is now happening is what I call runtime binding. So now with an Atlas API. I'm getting the information. I'm evaluating Ryan at that point in time. And the model is now computing it. The model is not predetermined. It could still have deterministic models also underneath it. And it is reacting in the moment. So that is the piece that people are afraid about is when it's reacting in the moment, is it going to be right or wrong? Right. So but it's the same thing even in a personalization. Right. In an ML model, you do not know whether it's right or wrong, right? It's experimentation, right? So that is why loop and feedback loops are very important in everything that we do. The key thing that I would say is trust.
RD: Yeah, right. We are in agreement there.
SV: And there are two pieces to it. We have always been a payment provider. We have made sure that we facilitate money movement for what matters. But there are two parts to it. One is driving demand. and the other was improving conversion. Obviously, as a payment partner, we partner with merchants on improving conversion. But it's equally important as to go upstream. And that's where what we feel with PayPal and with the transaction graphs and the commerce intelligence that we have, we are now able to seed it in the agentic interface so that, Ryan, you go to abcsite.com. We know that you are a male if you're given the consent and you're looking for size 10 shoes. So when you're searching, you only see size 10 shoes. That helps the merchant convert better. That is not historically a checkout conversion funnel. It's an upstream demand generation funnel. So that's kind of where we are trying to partner with merchants and how do we drive business revenue for them not just optimize the checkout flow.
RD: All right. Better shoes online. I love it. Yes.
[music]
Thank you for listening, everyone. I've been Ryan Donovan. I host the podcast, edit the blog here at Stack Overflow. If you have questions, concerns, comments, topics to cover, email me at podcast@stackoverflow.com. And if you want to reach out to me directly, you can find me on LinkedIn.
SV: I am Srini Venkatesan, CTO at PayPal. If you want to connect with me, reach me out to LinkedIn.
RD: All right. Thank you for listening, everyone, and we'll talk to you next time.
[outro music]