SPONSORED BY DISCOVER FINANCIAL SERVICES
On this sponsored episode of the podcast, Ben and Ryan chat with Paul Manning and Emanuele Pugliese of Discover Financial about the tech that goes into payments and the way they approach developer experience and architecture. They talk about domain-driven design, event-driven architecture, Kafka Streams, and how they leverage all that to build interesting FinTech products.
Episode notes
Go deeper into their tech at Technology at Discover.
If your interested in working on an event-driven architecture that uses domain-driven design within a financial organization, check out jobs.discover.com.
Connect with Paul on LinkedIn.
TRANSCRIPT
[intro music plays]
Ben Popper Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am your host, Ben Popper, Director of Content here at Stack Overflow, joined as I often am by my colleague and collaborator, Ryan Donovan. Hey, Ryan. How’re you doing?
Ryan Donovan I'm good, Ben. How’re you doing?
BP Doing well, thanks. So today we have a sponsored podcast brought to us by the fine folks over at Discover, and we're going to be chatting with them about some of the tech that goes into things like payments and the way they approach developer experience and architecture. We're going to be talking about domain-driven design, event-driven architecture, Kafka Streams, all those fun things that you can leverage when you're building interesting FinTech products, both from the customer experience as well as the experience of the folks on the inside who are building it. So today, let's welcome our guests: Paul and Emanuele. Welcome to the Stack Overflow Podcast. Both of you, hello.
Paul Manning Hello! Thank you. Good to be here.
Emanuele Pugliese Thank you.
BP So Paul, just really quickly, why don't you tell folks who you are, what it is you do day-to-day, and how would you define Discover, especially as it is in the UK where you are and maybe it's not quite as well known of a brand.
PM Cool. So, hello. I'm Paul Manning, I'm a Principal Application Engineer. I've worked in the payment industry now for nearly two decades, and I've been with Discover in the UK for over three and a half years. So I work in the digital payments domain and I do everything from delivering some of the business value stories, but also working across some of our dev teams for some of those bigger technical challenges– things like observability tracing. And then just last week I was doing a few internal presentations on chaos testing, so a nice mix of challenges and dev stuff. For Discover, as you said, in the US we're well known. So Discover Financial Services, we are a digital banking and payment services company, one of the most recognized brands within the US. We're one of the largest card issuers in the US; we issue the Discover Card, and then the banking side of the business does student, personal, home loans, checking, savings accounts, et cetera. And then we even operate the Discover Global Network, so there's millions of merchants and cash access locations, PULSE, which is a leading ATM debit network. And we've got Diners International, which is a huge global payments network with acceptance all around the world. And then as I mentioned, I'm based in the UK, so here we've got a really strong engineering presence. So although we are relatively smaller in terms of number of people, we're highly thought of. We've got our own hub in Farnborough in Hampshire where we're seen as a unit in our own right with kind of our own identity. And then working with the technical teams, we've got a really strong approach to upskilling our engineering workforce through collaboration, innovation, learning. We’ve formed some really good internal communities to help build our craft, improve our products, and grow as engineers. So we place a great emphasis on engineering talent, so it's a great place to work.
BP Terrific. Emanuele, would you like to give folks just a quick background on who you are and what you specialize in?
EP Yes. Okay, my name is Emanuele Pugliese and I've been working at Discover for about seven years. I work as an architect for the digital payment platform, and basically our mission is to enable all the cardholders to be able to make payments using their mobile devices and their apps, and this is our core area. In addition to what Paul has said, I would add that Discover gives the opportunity of working in different departments and learning a lot about the payment ecosystem. I started as a senior engineer on our data integration pipeline using big data and technologies such as Spark and HBase, and after that I worked as a tech lead on the dispute platform, mainly working on microservices and RabbitMQ, and finally on the digital payment platform using Kubernetes, Docker, and Kafka. Therefore, I want to say that for engineers working at Discover, we have got an amazing opportunity in using relevant technology as well as growing in our career.
BP Very cool.
RD So obviously Discover is a big player in the payment space, so what's the tech behind a global payment network like this?
PM So within the payment space we're doing a whole load of cloud native containerized microservices: Java, Spring, Spring Boot. We mix SQL and NoSQL data stores and we leverage a mixture of event-driven and RESTful APIs, so it's a really nice modern tech stack with a big, big emphasis and high importance on security, resilience, those kinds of things. And it's also worth adding that we've got a big emphasis for our teams for “You build it, you run it.” So our dev teams, they're all empowered to come up with solutions, new technologies, and ways of working that will really enable us to be the best that we can.
BP Awesome. And Emanuele, I know out of that tech stack, you focused a bit on event-driven architecture. Can you define for folks what that is and then maybe talk a little bit about how that's applied at Discover?
EP Yes, basically at Discover we use event-driven architecture and we combine it with Kafka Streams and domain-driven design as well. We think that these architecture patterns really complement each other, and to understand why, I would like to maybe start speaking about orchestration and the choreography and compare these two design patterns. In the case of orchestration, this is often used with asynchronous REST APIs to target specific backend services that are responsible for both processing requests and processing the response. In the case of an event in choreography, the backend services that process the request may be different from the backend service that are processing the response. This is a very key point that allows us to add multiple services and business capabilities over time, and those new capabilities can be run in parallel or contribute to the final response of our original request. Therefore, this becomes a flexible mechanism for evolving our design and the scaling services in different ways. Shall I go with an example?
BP Yeah, an example would be great.
EP Then let's assume we have a client that wants to enable a cardholder or cardholders to add their payments card on their mobile wallets. Then we decide to create a service to interface with the client, and then we create a card eligibility backend service to verify if the card is eligible to be added to the wallet, and finally another service, that is the virtual card service, to create a virtual card. The virtual card is then added to the wallet in place of the actual payment card for security reasons. To be able to orchestrate the two backend services, we decide to create a central orchestration service, and this is okay until we get a new client that has got slightly different requirements. Now we decide to, for example, change one of the APIs, like the card eligibility APIs. And then soon after, another 10 clients are coming in with slightly different requirements, and suddenly our orchestrator has become very complex and difficult to maintain. The advice here is, if we are dealing with multiple clients and the clients have different requirements, we should avoid creating a central orchestrator. To solve this problem, we can use events and choreography, and in this case, we could design the card eligibility service to emit a card eligibility provided event that the virtual card service can listen to. If the card is eligible, then the virtual card is added to the wallet and there is no need for a central orchestrator anymore. And at this point, the service can use a request response in the same pattern to correlate the different topics. Again, we need to bear in mind that the two topics have been populated by different backend services in this case. And the request response is a valuable pattern, but there is an even better pattern in many scenarios– it is Kafka Streams. Because events are stored in Kafka, each topic can be seen as a stream or table of events. Therefore, a Kafka Streams library can perform join and aggregation operations directly on topics alongside other operations. And in a simple use case scenario, we don't need to even maintain a data store or a database; Kafka Streams can do it for us. And this is why potentially Kafka Streams and event-driven architecture are sort of Lego blocks that allow you to build a better decoupled architecture, because you can plug in all these services in the middle and then design them properly. That’s an example, basically.
BP Yeah. So I think one of the things Ryan and I love to do when we're talking to folks is a little bit about the pros and cons.
PM Yeah, so I think, Emanuele, you mentioned some of those benefits, but one thing that struck me is when we do a proof of concept. And that's quite easy to get, there's so many tutorials and walkthroughs out there. You get those working and once you've high-fived each other because you've got something working, then when you try and scale it out, you start to spot those pain points that you need to think of sooner rather than later because they're only going to hurt the further you kick that can down the road. And something like sharing a schema between a producer and a consumer can seem quite straightforward to start off with, but once you start to scale that to tens and hundreds of consumers, that's a whole different problem. So if you've got a producing team wanting to change a schema, communicating that change out to those other teams can be really difficult. So some kind of central point source of truth to retrieve that schema is needed, and then you need to think how that’s going to evolve over time. Everything's about evolving architecture, so how does that evolve over time? And depending on how compatible your change is, that could influence how you have to upgrade your producers or your consumers and which ones go first. And then finally, don't forget if you are replaying old events, you can still need that old schema way after if you're replaying events to reconstruct some form of state. And then finally, you've got to think how does all that get through test to production? How is it rolled back if it goes wrong? With a CI/CD mindset, you want those producers and consumers to be able to release at any point, so getting all that testing is vital and getting that contract testing, much like in a RESTful world between your server and clients, is important as well.
RD So we talked about replaying and making sure you get the data, but what about when you retry actual events? You don't want to charge people twice, so how do you go about doing reliable retries?
PM Yeah, that's an interesting one and it's a very different world from the RESTful response. Because once those events are on your queue, your topic, your partition, the onus is on the consumer to handle those retries. So we looked from the two different types that to me stood out. There's a blocking retry whereby you go around and you try and reprocess that event, but you've got to be fully aware that no other events are going to be consumed on that thread until you've sorted that, so that comes with pros and cons. The other way we looked at it was, you've got these no blocking, asynchronous type retries, so you put it onto a different topic straight away and then you can take some time over that. You can have exponential back-offs, you can have max attempts at retries, and maybe you even have to produce that to a series of retry topics. Ultimately, there always becomes a point whereby you've got to put it onto some kind of dead letter topic, dead letter queue, and maybe some form of manual intervention is needed there to correct it and replay it if necessary. You can also look at the type of exception that caused that event. So something that might be retriable is maybe a transient network glitch. You get those but quite quickly that might fix itself and you can go and straight away replay that, and others are just inherently more complex. It's a lot easier with idempotent events as well; I'd say that's a massive thing. The more idempotent you can make things, definitely easier. And then finally you've got to design all that in mind, but it comes with quite an additional cognitive load on the developer to understand if you've got a complex retrying strategy. And also then, don't forget that you've got to have the observability, the monitoring, the alerting tools in place and test it as well, because you don't want to find out at three o'clock in the morning that something's gone wrong in production and then find your observability and your monitoring tools aren't up to scratch. So we found something like game days, for example, where you get everybody together, and for us that's UK, US, you get everyone in a virtual room and you play through some of those scenarios, and that's a great way of bringing those teams together and having a look. It's brilliant for fostering collaboration as well, so it's great.
BP Yeah, that's interesting. Ryan and I were on a call earlier today all about SRE and trying to install a culture where if folks build it, they should know that at some point they're going to be the ones on call. So when they're building it, get into that mindset of reliability from the very beginning. And I think to your point also, having these group activities where everybody gets to participate in that, to understand where some of the responsibility lies, can build that culture of understanding, “Well I can't just create this, chuck it over the wall and it's somebody else in operations’s problem then.” We all have to maintain responsibility sort of as a team for this.
EP Yeah. And I would like to add also that maybe from the same pattern point of view we could decide to implement idempotency as Paul was saying, but also the outbox pattern would be a very good way of avoiding reprocessing the same request multiple times. But going back to Kafka Streams, that allows us, if properly implemented, to use exactly one semantic, which means that we can prevent data loss and duplication, which really sits well in this use case scenario. There are different advantages. You reduce also the code footprint, but more importantly, exactly one semantic is something very difficult to achieve that can be achieved with Kafka Streams.
RD So we talked about the streams, but the other part of what y'all brought to this was about domain-driven design. Can you talk about maybe how that fits into it and if you have any sort of best practice tips for other people trying to approach domain-driven design?
EP Yes, we design our system reflecting the business domain requirements and what are the aggregates and microservices backing up those aggregates. And then what is really important once you have done that work is to be able also to document the design and the decisions that have been made for the current architecture and the target architecture. And I found it personally very difficult initially to understand how that could be done, and I have got a few tips on that. If you are trying to document the decision being made, there is the architectural decision record template– ADR. This is a template you can find online and it's about documenting the problem statement considering the options and the final decision that has been made. And on top of that, we want to document also the domain or subdomain. This requires a dedicated documentation that is more static. You can use the C4 model. This is another template you can find online and it is about documenting the context around your domain, the content as components, and potentially the code as well. And finally, once you have got all that documented, each incremental value, initiative, or project will require also an end-to-end documentation where you bring together all those sub-domains and then you provide clarity around the design and how this is impacting your system. These are the major tips I would have for documenting DDD.
BP Paul, how about you? Does this come up in your actual work with you and engineers and teams you work with? From the DDD perspective, what would you say has been important and valuable?
PM I would go with ADRs again. I'd reiterate what Emanuele just said about ADRs, I think they're absolutely brilliant. Literally just a couple of weeks ago I had someone who was talking to me and it reminded me of some work that I did about a year ago with some performance testing, and it was great that I knew exactly where that was. It's in the code. I could give them the GitHub link and it's got everything, all the challenges that I faced at the time, the rationale around some of the decisions that were made, and I could send that over and it's literally a snapshot in time. Brilliant. Use them.
BP So we've chatted a little bit about the benefits of DDD, EDA, and Kafka Streams. Are there challenges or cons associated with this as well? If people are considering this as maybe setting up the core of their next payment startup or inside of their business, what should they consider?
EP Yes, it's very important for architects to have engagement with all the stakeholders, including the business. We think that the business is the one providing requirements and knowing everything about what is needed to be done, but in reality it’s a collaboration effort. It's about discovering what the current solution does, what are the client’s requirements, and come up with a domain model. And there are different ways of doing that, but even storming and gathering everybody in one room and discussing those requirements and systems, that is challenging as well. You find out that there are a lot of unknowns, a lot of ideas, and this is where I think it was mentioned in the documentation that it's important we start putting down all those ideas, documenting the decision, and aggregating the solution using the documentation types I mentioned. I think that was one of the main challenges. Another challenge is that actually we need to think about the system as a system that keeps evolving, and this is why it is important to be able not to have a design up front, but break down the solution in a way that is scalable. And again, this is where I think event-driven really helps by being able to plug a new capability using the existing topics and events you have in the system. And yes, mainly it's a challenging architecture pattern, EDA. It's more complex compared to having to orchestrate or managing the state, but at the same time allows you to, if properly done, create generic events. Those events are reflecting the business domain and then you can leverage those events from different applications.
BP Right.
RD So that brings up an interesting point– how do you actually scale an event-driven architecture? With normal service stuff you just spawn more instances, more containers. Is there something else you have to do with event-driven architecture?
EP This brings us to the challenge of how you can scale Kafka, and there are different ways. For example, you can link clusters, but mainly it is important to understand that scalability using Kafka is connected to the number of partitions, and being able to create the right number of partitions is important. And also running the performance test against a production-like environment is essential. There is no solution out of the box there. It's all about trying your business service, test it, see how it reacts, and then understanding the batching strategy, fine-tuning Kafka is really essential there. And we cannot in most cases aim to infinite scalability, but instead we should aim to understand our requirements first and try to achieve those.
BP Yeah, I think that's a really important point you make. So Paul, I know this wasn't part of the script that we had prepared, but that's why we do chaos testing. So you were talking before we set the mics live about how you had been working on some of that with your team. Ryan and I love this topic. We've gone over it with folks from AWS and Netflix and a bunch of other places. How have you been doing chaos testing internally and what are you seeing as some of the benefits there? What cracks is it maybe exposing that you are going to want to address in the future, and how does it bring your team together?
PM Yeah, so it's interesting. I've only just got into it over the past few years, it was almost always on the side. As you see, it feels like it's becoming more and more important. At Discover, resiliency and all the operational aspects are vital so chaos testing perfectly feeds into that. And when you're running in cloud native world, you've got all those little network glitches coming up and going down, all those kinds of things, so I've really enjoyed getting in there. You can add in little proxies here and there to add in and simulate some latency. And then you throw in some performance testing on that, and then you see how does that look, and then you get your dev teams in the room and say, “Okay, cool. How's the operational and the monitoring side looking with all this additional latency going on? Can you find out what the problem is?” We haven't done it yet, but one of my plans is to actually have a malicious actor in the room who's there tearing down pods, killing off partitions, taking nodes down, that sort of stuff. Let's see how evil they could be and how quickly you can identify and get over those problems.
RD That's right, have somebody in the room engineering the chaos for you.
PM Yeah, an evil person. That sounds good. I like the sound of that.
EP No, it is also very important to start doing and thinking about the chaos test at the start of your project. Even if you go live with very rudimental service, still you can test the recovery strategy. You can still try to bring down the network, bring down the pod, and see what is happening to your system. And it is really helping engineers to get accustomed to the idea that the application network and the cluster can go down, and just start thinking on how we can address those issues, then failover strategies, active-active, active-passive configuration, and it's a fundamental tool that we should make use of as soon as possible.
BP Very cool.
[music plays]
BP All right, everybody. Thanks again for listening. I am Ben Popper, Director of Content here at Stack Overflow. You can always find me on Twitter, or X as it's now called. You can find me on X @BenPopper. You can email us with questions or suggestions about the podcast, podcast@stackoverflow.com. And if you like the show, do me a favor, leave us a rating and a review. It really helps.
RD I'm Ryan Donovan. I edit the blog here at Stack Overflow. You can find it at stackoverflow.blog, and you can reach out to me on the former bird site @RThorDonovan.
PM I'm Paul Manning. You can reach me on LinkedIn. And also if you get a chance, check out jobs.discover.com.
EP I’m Emanuele Pugliese. I'm an architect at Discover Financial Services.
BP All right, and we'll make sure to put the link in the show notes. Thanks for listening, everybody, and we will talk to you soon.
[outro music plays]