ROCm is AMD's unified, open-source software layer and toolchain that allows developers to program on AMD's hardware accelerators.
Connect with Anush on X.
Congrats to Populist badge winner Blitz for winning the badge on their answer to Is there a way to view your apps dot entitlements file from your app’s IPA archive?.
TRANSCRIPT
(STACK OVERFLOW INTRO MUSIC)
Ryan Donovan: Hello and welcome to the Stack Overflow podcast, a place to talk all things software and technology. My name is Ryan Donovan, and today we're talking about the programming on the GPU. And not only just programming on it, just doing it with AI. My guest for that is Anush Elangovan, who is the VP of Software over at AMD. So welcome to the show, Anush.
Anush Elangovan: Thanks for having me, Ryan.
Ryan: My pleasure. Before we get into today's topic, can you tell us a little bit about how you got into software and technology?
Anush: Well, that's a very long story, but...
Ryan: Tl;dr. Tl;dr.
Anush: (Laughs) Tl;dr. I started programming when I was ten or something like that, back in India, and did a lot of computer graphics back in the day. Then did my computer science, focused on networking, network security, and then that slowly led to operating systems, worked on Chrome and Chrome OS, worked on gesture recognition. And then about ten years ago we started not.ai. That became... 13 years ago. That became... It was focused on ML compilers as AI was taking off, and we were building software that could adapt and be applied to various different accelerators. And then AMD acquired us about three years ago. And we've become, like, the core software engine for AMD.
Ryan: AMD was sort of originally known as a CPU provider, and it's getting into making GPUs and integrated CPU, GPU assets. With the GPU, you need to be able to program on the GPU. And that's where AMD has ROCm as a direct, general purpose GPU programming language, right?
Anush: Yes.
Ryan: Can you tell us a little bit about ROCm and how it works?
Anush: So ROCm is AMD's software layer. And like you said, AMD grew from CPUs, GPUs, FPGAs with Xilinx, FPGAs. But each piece kind of had, like, a board support package. So it was something to unlock hardware. It wasn't a platform that stayed on top of all of these hardware accelerators and components that we had. So ROCm has become this unified toolchain and layer that allows for customers to program on our hardware. And it looks unified as much as, you know, from, like, toolchains, debuggers, et cetera. But it provides you the core building blocks for you to build and innovate on. And all of it is open source. That's the key part of it. Which is different from Cuda, et cetera, right? Where. We want the ecosystem to flourish. So, that's a core philosophy that all the source code is open. And then it allows you to iterate fast, right? You're not limited by the speed at which we can provide you support on some closed-source binaries. We want you to see what the source is, and we want you to innovate and then build on top of it. So that's the core of ROCm.
Ryan: We love open source here at Stack Overflow. But obviously the ROCm language and toolchain has to fit with the AMD chip designs. How much leeway is there for open source developers to, you know, push their own features or fork the code?
Anush: As of about a year ago, we started this effort to not just make source code open, make the entire development process open. Right? So today, if you found an issue, or you had a bug, or you want to add a feature, you can go to ROCm GitHub, make the feature, submit it, and it'll get rolled into tomorrow's nightly. It is a pure open source project. Including the build process and everything included with it.
Ryan: You know, I'm not super familiar with GPU programming. I have seen a little bit of sort of CPU programming. How much sort of direct hardware access are you doing there?
Anush: It's like, very direct. It is all like, direct to ISA, right? One of the advantages that AMD has is, we even publish the ISA spec, right?
Ryan: Mhm.
Anush: I know in CPU land, it's like, given that you know, like, okay, what are the x86 instructions? So you can write an assembler, you can write a compiler. GPUs are not like that, other than for AMD GPUs, right? So we actually document the ISA for each of the GPUs that we launch. And we just launched the MI455 last month. And as soon as it was launched, we launched the ISA. So anyone in the world can pick up the compilers, pick up the tools, program to our GPUs, the GPU programming itself. If we go a little bit more into the technical details, it's a little different. Where, it's called SIMD. It's like a group of threads. So you're programming, you have a very highly parallelizable groups of threads, thousands and thousands of threads, doing kind of similar-ish work. So it's like, good for heavy parallelism. Versus CPUs and processes and threads. You could have more fungibility of like, what runs, right? You could have a SQL server and a web server and something else, right? Whereas in a GPU, you're usually saying, oh, I'm doing a matrix multiply, and I'm going to decompose the matrix like this. And here a group of 1000 threads that are going to compute these individual components. And then I'm going to group it and run on, like, 500, and group it and reduce it to like, 200, or, you know, and then find an answer, right? So it's good for heavily parallelizable work that is predominantly seen in HPC workload, scientific workloads, and of course, AI workloads.
Ryan: The way I sort of think about it is, CPU has a lot of big compute chunks. GPU has very small compute chunks, right?
Anush: Yeah. But also it's one way, because your instruction caches and your instructions are shared between all of the thing, right? Like, it's the same kind of code that's going through all of these threads. But GPUs are increasingly getting more sophisticated and they're a realm of, like, having something called warp specialization, for example, right? So you're doing conditional flow control inside some of these things. And then that's starting to look, like, CPU-ish, right? I wouldn't say it's like, generally that way, but you're starting to do like, okay, this group, I need to do this, and you need to specialize this, and that can finish earlier. And then I need to move some memory. And so. It's interesting. As a computer scientist, it's fascinating, and it's a challenging area to be in.
Ryan: I do understand it is very challenging. I've seen some sort of SIMD CPU stuff, and it's very raw access, right?
Anush: Mhm.
Ryan: It seems like it's a little bit unapproachable, even for your standard developers. Is that right?
Anush: So the way I would think of it is like, it is increasingly changed, right? Like, back when you had to go into CUDA or ROCm, like, hip kernels, and like, program in low-level C, it was a challenge, because you already had that barrier that you had to get an understanding of a new machine architecture and program to it and move memory and compute. And there were only, like, few experts that were there. But over time, what's happened is, like, the abstraction levels have increased to bring all of the tools up to a pythonic DSL, right? So today you can write GPU kernels in Triton, and that makes it really easy. And so we have something, Triton, there's something called DSL that provides you pythonic... Like, it feels like you're writing a Python program. You are writing a Python program, but you're writing a Python program for a SIMD machine and SIMD kernels. And we've brought GPU programming up a level into where ML practitioners live in Pythonland, and bridge that gap easily, so that it's relatively easy for you to program on a GPU. But that's just one part of the story. The past six months, eight months, agentic AI has just come in and like, further reduced the barrier, right? Like, you could literally just sit on Claude code and be able to bust out a kernel. If you just specify what the kernel should do, put it into it and serve on VLM or SD lang. And it's that easy, right? And that's part of our strategy as to, you know, we had open, but now we have AI-enabled, right? So we have now gotten all of our AI-enabled hooks so that ROCm is not just powering the top labs with AI, powering AI, but it's also being completely rewritten with AI, right? Like, you can actually have, you know, performance optimizers that are LLM-driven. And one of the coolest features of this is that, since all of ROCm is open source, all of the frontier models know ROCm as their base language. Because it's all open. So they could parse it, they understand the specs, they do this, they know what the output is, and the corpus is just continuing to increase. Whereas in a closed system you don't see all of that, right? So, frontier models can learn the input and output of CUDA 12.1, right? That's fine. But CUDA 13 we add something, and it has to learn a new thing from a black box, which is increasingly getting good with examples and all that. But for ROCm, the moment you submit a PR, it's like, those are tokens available for a frontier model to learn, right? It's either learned in the pre-training or post-training, it can find it. And we're seeing an immense advantage to acceleration of the ability to program on AMD GPUs, because of both, it's open and it's natively AI.
Ryan: And I think there's a couple things there. I know that GPU programming, hardware programming in general, has been sort of the domain of niche programmers. Because it's a very specific, you know, very low-level, very close to the silicon kind of programming. I sort of imagine people getting a little nervous about giving that over to AI.
Anush: Mhm. (Laughs)
Ryan: But the other thing, it sounds like you have sort of more abstractions between the silicon and the programmer that I wouldn't have expected as much.
Anush: The way I look at it is it's not more abstractions. It's just we're giving you different levels at which you can tap into the system, right? It's like a layered cake. And so, if you want to be in the basement and program in assembly, you're welcome to do that, right? Most people want to hang out in the living room and... (Laughs) And like, it's comfy. I got my Python, I got this, I got that. Or for some folks, it's like, okay, I want to be up on the rooftop and all I just want is like, you know, chill and just get through, right? So it's a layer cake, but we allow you to tap into any layer that is most comfortable for you. And increasingly, these layers are getting mushed because AI allows you to become an expert in a layer that you didn't have to be, right? Like, I can actually go into the bowels of the system and be like, reprogram this GPU kernel. And as long as I have the higher-level principles and the formulation of the task, someone just built an elevator in the building now.
Ryan: (Laughs)
Anush: So I can just be like, oh, I'm just going to go to building the basement and go check out what's the problem. And then I come back and I just chill out on the top rooftop and, you know, have my...
Ryan: If you're going down in the basement, how much of that is specific hardware configuration dependent? Does that apply towards every AMD configuration?
Anush: The way I look at it, down in the basement, we have the water heater, we have the electrical system. We have the plumbing. (Laughs)
Ryan: (Laughs)
Anush: So, sure, there may be, like, the CPU, GPU, and NPU, different things doing different tasks, but we give you the ability to go in and look at the details. Our basement is accessible, let's put it that way, compared to competition, right? Our thing is like, it's built for hackers, it's built for developers, it's built for sovereign and airgapped installs, where you're like, okay, what's in the basement? I just hear a thumping noise every night. (Laughs)
Ryan: (Laughs) Right, right, right.
Anush: Going along with the basement analogy.
Ryan: Yeah, yeah. For the person who's on the rooftop using AI for this, what are the sort of things that agentic ROCm coding enables for them?
Anush: On the rooftop it's about ease of use. It's about going to the outcome, right? Like, hey, I just want to chill out and I want to trade on my e-trade or whatever the thing is, or I have to plan a family vacation, right? Like, you're looking at outcomes. You don't care as long as there's like, you know, your basic things are working, right? So what we provide with ROCm is like, the ability to like, hook in at every level. But at the end, if there's no meaningful value built for you to use AI on AMD platforms, it's meaningless, right? So, taking an example of that, like, we have powerful products that are increasingly very delightful to use, right? And so we start with client and laptops. Strix Halo for the past three years has been the leading AI inference local inference story. Right. It's got 128 gigs of RAM. So you can run very large models on a laptop form factor, right? That's my primary laptop. And so you get to the outcome, which is like, I can now use AI without a Claude subscription. You can still use it for frontier intelligence, but that's what we enable with ROCm on the client side. But then we have on the desktop where you're doing gaming, or you're doing some other advanced simulations, and those need AI things, right? And so we provide the ability to inference AI in the desktop. And then, when you get to the cloud, that's when you know, at scale, all of the frontier models now, currently, or most of them, are currently using AMD at scale. And this is like, tens and thousands of GPUs that are in data centers to provide the intelligence that you've come to use on ChatGPT or Claude or Gemini, right?
Ryan: Is it possible to access the GPU without using a GPU programming language?
Anush: As much as you can do with a CPU. So. For a CPU, you could write machine code, and, in assembly. And use an assembler to just generate it in bit code. And you have a loader to load it. You can do exactly that on a GPU. On our GPUs. Other GPUs, you got to throw it into the basement and hope it kind of like... (Laughs) Right?
Ryan: Right. Yeah, yeah. Because since the rise of GPUs, mostly for gaming, and early days, like, I don't think I thought about it as a separate thing, just like, here's the little things that go to it. But it does, you know, seem like there's both a use case story and an optimization story with touching the GPU directly.
Anush: In the end, it's a programmable machine, right? It's got, like, classic von Neumann architecture. There's a compute, there's a load, there's a store and there's a loop. (Laughs)
Ryan: Yeah, yeah, yeah.
Anush: And so, it's relatively straightforward from that perspective. But you had referred to it, right? Initially it was thought of like a co-processor and accelerator. But if you looked at it in the CPU era, we needed the 486 and the 487, which was the co-processor that did math stuff, right? And now you had coprocessors for graphics, which is what GPUs were. And there was an attempt to do physics as a separate co-processor, right? If you remember. But then, GPUs got increasingly general purpose that they started to do physics. They started to do simulations, they started to do other things. And then it tripped on AI. And they're like, oh, this is the best thing since sliced bread. And now I'm like, if you look at market share-wise, the GPUs being used for AI is like, so much higher than GPUs actually being used for graphics, which is where it started.
Ryan: Yeah. Anybody who's tried to buy a graphics card recently has understood the market pressures on it. (Laughs)
Anush: Yeah. And then the components that go into it have their own story line, right? Like, memory and others.
Ryan: How's AMD sort of changing how they're thinking about AI? Are you programming chip designs using AI? Are you pushing PRs to the rocket base using AI?
Anush: About six months ago, it was like, almost mandatory that every engineer is starting to use AI, right? And it's not as an assistant, it's like, it is an agent, right? So we are increasingly getting agentic. So the software side is like, pretty quantifiable, right? There's lines of code, there's outcomes. I've generated more code in the last 4 to 5 months than in the past, you know, a lifetime of my GitHub, right? I think, pretty significant. So, code generation is no longer a metric. It's just like, okay, you tell it to generate 10,000 lines of code and you get 10,000 lines of code, right? But then now you have to think of, like, what's next? And like, agentic harnesses. Harnesses for software, harnesses for hardware is different. And now we have to think of, like, how do we pull in hardware validation in a way that we can leverage these agentic loops, right? Like, can we do better power performance analysis? Can we do better processes or nodes in terms of like, shrinking the die, et cetera, right? So, I think pretty much everyone at AMD now is like, agentic. And so, super exciting because that's an accelerant for AMD. And we could go from a, hey, we're tracking where the leader was going to. Like, now we can actually skate to where the puck is going, because now we don't need to go through the same discovery process. And now with AI, it's like, oh, we don't have a job scheduler. Great. We're going to write a job scheduler. Two months later, it's in production on 2,000 GPUs and move fast, right? And that's why I repeat, like, speed is the moat. There's nothing else that can keep you competitive in the world. It's just about how fast you can move and execute. And you may fail, but if you're fast, you're going to recover, right? And go forward.
Ryan: That's right. Fail fast. So, you talked about the feedback loops and iterating. With software, that's definitely faster and easier. With hardware there's a little difficulty to that because, you know, the actual hardware is a real physical thing that has to be manufactured, tested.
Anush: We've gone through, like, a very tightly integrated development loop with hardware teams, right?
Mm.
And of course, we have simulators, we have emulators, and then we have final silicon. We are getting to a point where we are... A year before tapeout, the software is done. Like, two years ago, AMD's like ROCm, we'd be like, oh, we released a product and we're going to optimize it over the next three months, something like that, right? Now, the product has been optimized, like, six months before tapeout, right? Like, before the silicon actually just leaves, we've already finished optimizing. So by the time the silicon comes back, we're like, ready to, like, hey, just press the button and it should run this, you know, signal integrity issues, validate that the silicon is in the right spot. There are some tolerances of voltages and power margins that we have to tweak. That's all like, non-deterministic stuff that we just have to deal with as silicon, right? So. Then you're like, okay, everything else is good to go, right? While we've already given that we've already finished the next generation and the next next generation, right? Like, the next generation is being validated. So, it's been, I don't like to use the word too much, but, you know, shifting left as much, but...
Ryan: (Laughs)
Anush: So compressed that it's gone through, you know... (Laughs) Through the left margin. But, yeah. Right now my recommendation is like, anyone from the hardware team, like, the moment they think of an idea, the initiation of the idea is, like, you should have it, something in software and the software is running, right? It's like, let's add an instruction. Great. The first action is like, add the instruction plus the software, and that's your MVP of even thinking of, like, am I going to add this instruction or not? Right? Which means software and hardware is, it's just one. Like, from an engineering standpoint, you treat it as one. It's like, I make these two things come together and move that ball forward. Then the timelines dilate, right? And then you say, okay, hardware takes this much time to, you know, physics and all that good stuff. And then there's, the software is like, reality of, like, you know, the non-determinism of the physics. And then you've got to build from that. And then there is ongoing support of platforms that we've shipped five years ago that we still want to support. So, software has to live on in perpetuity of the lifetime of the product, and then they catch back up in the next iteration of the delivery.
Ryan: Yeah. My understanding of emulation is you sort of map the capabilities of the thing you have to whatever hardware already exists. Are there cases where you can't emulate future hardware because of the capabilities of the current hardware you're working on?
Anush: Yeah. I think emulation is a very fantastic job that you can write immense amounts of PhD theses too, right? but if you back it up a level at a product and a business standpoint, you think of it as in the simulation/emulation curve, right? And actual silicon. You start with like, hey, can I do functional verification? Right? Like, if I just had something that had the ISAs, it just says the ISAs work that way, but there's no performance metric. It's like, it's good, your program runs and it does the right thing, if those instructions were implemented, right? Then you get, in the emulation side, you may do something that's like, emulated on CPU, emulated on a GPU itself, like what you talked about, right? Which is like, hey, let's emulate this on a prior generation hardware. So then you do what you asked for, which is like, hey, tensor data mover was added. So now we need to emulate tensor data mover in a prior generation hardware. Which is okay. But then there's also, like, okay, can we actually emulate the silicon that's going to get taped out? But we have to do it in an FPGA. But then we need, like, big FPGAs and collection of FPGAs, so that you simulate a full chip on like, a cluster of 16 FPGAs or whatever it is. Right? And then you're emulating the next-generation hardware with something that is expensive and strung together in some form. But it gives you some confidence that, okay, this thing will work, right? And then you get to the next phase, which is like, actual silicon, and you're de-risking the silicon bits.
Ryan: I think, like you said, the software and hardware are converging. Because on some level, the hardware is the same sort of logic gates that you can do in math. So, I wanted to sort of talk about how your team is using AI. I've talked to a few folks in the big enterprise teams running complex things, and there's a lot of really interesting sort of snout to tail agentic harnesses going on from design to observability. How much of your SDLC has gone agentic?
Anush: I'd say a pretty significant portion. But not the entire loop. So the funnel is... The gates are open, right? Like, people are like, rushing in, and they're like, hey, I'm doing this. But then, the outcomes from being agentic haven't landed yet, right? In the sense that, I'm running a bunch of agents right now. They're doing everything from, like, a formally verified rust operating system to testing out the latest, I think Qwen 3.8 came out last night or this morning. I want to check the performance. All of those are like, individual agents that are running. I have like 6 or 8 agents that are doing like, long-running, complex tasks that kicked off. But the key for a team is that you're going to see, like, what I call, like, the K-shape of divergence, right? Like, you'll get the top arm that people just have taken off. They're just like, generating so much code. They're generating so much, like, outcomes, right? They're just like, okay, we don't have this. I'm going to do this. I'm going to ship this. And it's only like ten, 20 people or 10, 20% of your team. And they're just like, off on the rocket ship, right? But then there's like, the good general mass that's like, okay, this is agenetic. I'm seeing the benefits. I can do it in this use case. I can do it in this... They're like, the living room area, right? If you... (Laughs) Like, the...
Ryan: (Laughs) Right.
Anush: But then there's, like, another chunk that, you know, you have, as leaders have to help, right? Like, you have to give them the comfort, the tooling, and the ability to upskill and reskill on how to use agentic flows, right? So there's... There is a theory of like, where people fall. And, you know, I think it's humans, you're going to have that at all times, right? There's going to be people that are super interested in it, people that are, like, skeptical. And we have that spread just like every other place, right? But I think as leaders, we want to try to tend up, into that up and right corner. And we want to be supportive of bringing everyone along for that journey to get there.
Ryan: Right. And you want to be able to get those trust signals, and sort of internal verifications, and guardrails, and all that sort of thing. For example, the testing, the current model. I wonder what kind of checks and controls you have on a process like that to make sure that, when it spits out numbers at the end, you can believe them.
Anush: The reason why AI is so successful with SDLC and models, and this is that it's measurable, right? So, all I'm going to do is like, okay. Qwen 3.8, ensure that it is... The GSM 8K numbers are consistent with what you'd see with the previous model or some other model. So there are agent harnesses that are really, really good at measuring the quality of the models and the quality at which it's serving. So even if you were to do some quantization, it makes it relatively easy for you to, like, swap that and say, hey, this quantization, the degradation of the perplexity is A or B, right? And that may be acceptable. Because you're now serving at 10,000 tokens more. Or whatever it is. It's a trade off. It's a system design trade off. Agentic AI is like, really transformative, right? And we're increasingly getting to a point where what was previously... It wasn't in the realm of like, okay, why would you do that? That would take a long time, right? It's like, you know, I rewrote the Windows driver for AMD hardware. Like, six months ago, I'd be like, no, that's a large team effort. Multi year. We got to do this... But now it's like, oh, I'm going to go do this now because I have an itch to scratch. So the barrier to entry for these large projects is like, drastically falling. Which means now you can actually take on, and spread your wings wide, to say, hey, I want to do more of this. And so, as users, you start, like, getting a broad ability to like, spread yourselves into areas that you wouldn't have been able to do in the past. But it's very exciting time. And I think I'm very pleased to be in this takeoff. And I think the impact on, like, not just in technology, like, humanity as a whole, will be pretty significant, because humans, hopefully, would be able to spend more time on things that require human intuition, human understanding, and let the grunt work of like, file my taxes or do this...
Ryan: Yeah, that's...
Anush: Can get agenetic, right? Which, previously, you couldn't give it at that level. It had to be like, much more automated things level, right? Which is fine. It's like, ring alarm when this happens or that happens, right? That's like, that's one on one. But now you can actually give agents personas to go and be, like, your chief of staff. Or go do my finance and watch my finance and do something. And then you uplevel it, where they all come back to you and you can operate at a higher level.
(MUSIC PLAYS)
Ryan: Well, it's that time of the show where we shout out somebody who came onto Stack Overflow, dropped some knowledge, shared some curiosity, and earned themselves a badge. Today we're shouting out the winner of a populist badge, somebody who dropped an answer that was so good it outscored the accepted answer. Congrats to Blitz for answering, is there a way to view your app's .entitlements file from your app's IP archive? If you're curious about that, we'll have the answer for you in the show notes.
I'm Ryan Donovan, I edit the blog, host the podcast here at Stack Overflow. If you have questions, concerns, topics to cover, you can email me at podcast@stackoverflow.com. And if you want to reach out to me directly, you can find me on LinkedIn.
Anush: I'm Anush Elangovan, and thank you, Ryan, for having me. And you can find me on X. My handle is @AnushElangovan, and you can find out more about AMD and ROCm at amd.com.
Ryan: Alright. Thanks for listening, everyone, and we'll talk to you next time.
(MUSIC PLAYS)