SPONSORED BY INTEL
Ryan and Ben chat with Raymond Lo, AI software evangelist at Intel, about the AI PC, the software that powers AI breakthroughs, and optimizing hardware and software in unison to improve generative AI performance. Bonus: what’s the difference between a GPU optimized for graphics and a VPU or NPU optimized for AI?
Episode notes
If you’re interested in trying any of the demos that Raymond talked about, check out Intel’s OpenVINO notebooks.
Learn more about Intel’s Edge AI resources here.
Raymond previously wrote about enhancing image and video resolution using OpenVINO.
You can reach out to Raymond Lo on LinkedIn.
Congrats to Lifeboat badge winner, Andrey Korneyev, for saving the question, How can I delete specific nodes from an XElement?
TRANSCRIPT
[intro music plays]
Ben Popper Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am your host, Ben Popper, Director of Content here at Stack Overflow, joined as I often am by my colleague and collaborator, Ryan Donovan, Editor of our blog and maestro of our newsletter. Today is a sponsored episode brought to you by Intel, and we're going to be talking with someone who has worked with us before: Raymond Lo. He wrote a great piece for the blog, and we're going to be chatting about a few things related to AI and ML, and then specifically some new things that Intel has been bringing to the market and wants to focus on: the AI PC and the NPU. Not the GPU, not the VPU– the NPU. So we're excited to have the conversation, always a delight to chat with Raymond who's bursting with energy and ideas. So without further ado, Raymond, welcome to the Stack Overflow Podcast.
Raymond Lo Great to talk to you again, Ben and Ryan.
BP So you did a blog post with us. It was about upscaling graphics resolution and sort of vision-related ML. Since then, Gen AI has become the hot topic. Has your focus changed or broadened to support this?
RL So we made an observation almost early last year. We noticed Stable Diffusion where people started generating very beautiful images, and then we started also seeing the chatbot. People started using tools that can communicate with a computer very differently. I think when I first started doing chatbots many years ago it was pretty not good. I think that's the word. It just didn’t answer the question. But today I think Gen AI, especially with the large language model, provides a lot better outcomes. So early last year, we started working on it, and then later that year, we even released a lot more example code for people to try, and now today we have AI PC. It runs some of these workloads very well.
BP So can you just define for the listener, what do you mean by AI PC?
RL So a fun story, I think the name came from our CEO Pat. It was just the time before the innovation when we looked into the hardware, we started to see the new era where computers are a lot smarter. And why it's called AI PC, I think there's a very long story, I'm not going to disclose that today. It has to do with our initiative of bringing these kinds of capabilities onto a silicon that actually can generalize on AI. So what we mean by AI, it means something much more intelligent. The chatbot actually runs on those laptops, with some limitations of course. And then there's some things that you can run in the background, so that it’s always, for example, doing noise reductions, cancellations, or speech recognition. Those can run in the background continuously to support you, or, let's say, this conversation if you want to transcribe this or you want to have a draft from it. So that sort of thing can help a lot with productivity. That's where the AI PC stuff kicks in. It is the intelligence of the PC.
BP Got it. And just to clarify there, I liked what you said. We're talking about how you can help your customers, whether they be big enterprises or consumers at home trying to run Stable Diffusion. How can you give them more power, more horsepower, more flexibility at the silicon level, not necessarily just at the software level. So Ryan, go ahead.
Ryan Donovan I think when we talk to people about Gen AI, they're always doing the infrastructure on cloud computing. Is there a benefit to kind of moving that workload to the local consumer level hardware?
RL So it really depends on the workload, again. The advantage often I see is that it's available, there's no wait time. So even the connection problem you don't have to deal with. The privacy, the security part, again, rementioning the chatbot, sometimes you just want to ask a question like, “Look up the document here and explain it to me,” and that document could be confidential, so you don't have to worry about those concerns. So there are a set of workloads that works extremely well, especially for enterprise use for local. And then especially if, let's say, I already bought the machine, by the way. I have one of those AI PCs with me. But when you have millions of this in the market everyone has the same capability and it makes the deployment a lot more scalable too. So, for example, you have a hundred staff that want to run runTools. Either you buy some hours or you’re just like, “Okay, here's an app that we built.” So that creates that new, “I don't want to go back to the cell phone moment,” when the cell phone is fast enough to do something. Back then you had to go to the computer to browse the web, and all of a sudden it's like, “Wait, I'm browsing the web now, but it's kind of a little bit slow, a little bit downgraded, lower, smaller size.” Then all of a sudden now I hear everyone, “Why are you not going online with your phone?” So personally, why are we not using a big computer to do browsing sometimes? Convenience. So there's a lot of that that will kick in at this point.
BP Just to sort of educate myself I went through the process of, “Okay, how do you get Stable Diffusion up and running on a Mac?” You’ve got to do some work in Terminal, you’ve got to have Homebrew, you’ve got to figure a few things out, and same thing for LLaMa 2. And so the level of comfort that I have with that stuff is minimal. I'm not a great programmer, if I could even call myself that at all, but I mucked around in the Terminal enough to sort of work my way through it. It is interesting to me what you said that maybe somebody will build sort of a front end UI that’s like, “Oh, you want to try image generation? You want to try chat? Here's your UI. Pick the model that you want, we’ll download it in the background, and then we'll take it from there.” I think that could be really interesting. That's what ChatGPT did really well. They gave a great user interface to what's happening in the background. But also now, I hadn't even thought about, as you said, that maybe I should get a PC so that I can run some of these workloads at home, because it's not a great experience on my work laptop. That wasn't why I purchased it. I purchased it to browse the web and because of its speed when I'm doing simple tasks, and its size and weight. Of course, then I could say it's for work and it’d also probably be a pretty good gaming PC. I don't know, does an NPU work like a GPU or no?
RL No, but the GPU, actually, we almost forgot that they're actually very powerful. So they rebranded it as Intel Arc. I got the machine two days ago. Actually some of the workload runs even very well on GPUs because they are high throughput. I always position the silicon this way– the CPU is generic, anything can run like general computing. Then you have the GPU for graphics, but also very good for high throughput. Some AI tasks like image generation that you talk about are amazing on the GPU. NPU often I think is low power. It's like background tasks, so pre-processing of, let's say, background removal for images is perfect for the NPU. They're very low power, so if you don't want to drain your battery, use that. And the magic is that if you use OpenVINO, you can combine them all and use them interchangeably, or at the same time. And that would get pretty hot. It’s like, “I’ve got to use all the resources,” but you have the option. I think that's the nice thing about it. This is like a hybrid car but with three different engines. It's kind of cool.
RD Like Ben, I tried running Stable Diffusion, but as Ben was saying, there's a few front ends to large language models, and I tried one and it's remarkably slow how slow it generates on my regular computer. What does the AI PC have that could speed that up?
RL So there's twofold to the speeding. One is that the slowness often comes from unoptimized code that runs on the CPU only and maybe single threaded. With OpenVINO– again, full disclosure, I’m the OpenVINO evangelist. I've been working on this for the last few years.
BP I know we talked about this and it was in the article, but just for folks who are listening, give them the high level. What is OpenVINO so when we're talking about it, they have that context?
RL We can think of OpenVINO as the tool for you to accelerate your inference workload. For example, object detection, image classification, all the classical ones. But today, the last couple of years, we advanced to doing transformer work. So if you think about Whisper, Gen AI-related like Stable Diffusion that we talked about, we accelerate those to run on CPUs, GPUs, and also the NPU. It’s a tool that enables you to take this model and run it extremely well, much better than just the native way, on the Intel silicon. And that's what OpenVINO is great for.
BP Gotcha. Open toolkit facilitating the optimization of deep learning models and an inference engine. Cool. So sorry, please continue from where you left off.
RL So why it’s faster is actually, if you were running that poor experience, then now if you try what we are working on today, I see dramatic speedup. So for example, if something takes one minute, it’s going to do five seconds. It’s dramatic. And guess what? It's possible. So for example, we have a much better model, like LCM– Latent Consistency Model, for example. I can generate an image in about four to five seconds right now, very beautiful images. I can show you a demo right after, by the way. And then the reason it runs that fast is it is tapping into the GPU as well. The GPU has a very good workload, and it can crunch those pictures very quickly. And that's how we’re actually getting this a lot faster– utilizing the right resource for the right task. So again, high workload but on the GPU and then it would just crunch it through.
BP Right. Like you said, you got three engines and they're good for different things, and software that would let it know where to go when.
RD OpenVINO, does it work with the traditional ML pipeline stuff like PyTorch and TensorFlow, or does it replace it?
RL So from a usability perspective, if you're a developer today, yes, we'll just take the model directly. We'll do the conversion for you and it will get into the OpenVINO runtime. So this is what we call the engine sometimes back in the old days. So the runtime will take almost most of the famous framework. We also have TensorFlow, we have PaddlePaddle even from China, actually. We also support some of the little bit older ones, Caffe and et cetera. So yes, it will work directly with those models. And then I think to you as a developer, one of the famous ones is called Automatic1111. I think everyone may have heard about that in Stable Diffusion. We even go all the way down to what the community was doing and then modify our runtime to support it, so we even make it almost backwards compatible. So there's a compile flag to just make it like a backend for PyTorch to use OpenVINO. So in short, I'm a developer, I'm not going to easy sell. You have to read the documentation, everyone, a little bit there. You have to read, you have to do the homework. If you read the homework and you do the example code we do, you'll see all the examples, a hundred examples. You'll figure it out. And we're trying to make it better. If there's any feedback, let us know.
BP So I wanted to ask a little bit about what's new in the GPU and then talk a little bit about what is an NPU and how is that a sort of evolution of a VPU. I just want to say that it's really interesting for me– this is just a side note, but as it turns out, I was a reporter focused on the world of drones, and I wrote about VPUs back in March of 2016. DJI was working with Movidius, which I believe was eventually acquired by Intel, and at that point in time, they were saying, “This is a VPU. It's for doing vision processing on the edge so that the drone can avoid obstacles.” And at that time, the VPU was a new thing they were trying to coin. It didn't exist in the world until then. Now we're talking about how you've taken the VPU and evolved that to an NPU. So kind of a cool throughline, but please go ahead.
RL So the story where they did the same thing, almost. Movidius. So under the hood, if you look at the actual material I'm working on, it's the same VPU kind of material. That's why when we released this NPU today, we do have some support immediately. So that was the same initiative that you talked about very closely going back to here. Now back then it was a USB stick. I think everyone may have heard about the Compute Stick. There's a NPU Compute Stick. So we now directly integrate it into the silicon and that's some of the big changes that you may have seen. So to talk about that, I think that initiative and how this comes together, there's some history that's not very well known because in between we already had some of the integration happen with the Atom processor. So on the edge, there's an actual hardware that we integrated in the past to demonstrate this capability, but today is the first time we bring it to the consumer. ‘The consumer’ is everyone's laptop having that NPU. I think the differentiator for how I see this for people as a developer is that it gives you the leg up on why the drone wanted a VPU but not put a GPU on it. I think there's an obvious answer to everyone: power. It will just run out of battery before you lift off, so there's an obvious reason. And then for the consumer level, it's the same reason. You want a laptop that runs 12 hours. The last thing you want is that I make a phone call with my wife and then two minutes later it runs out of battery. It's like, “What are we doing? Oh, it's a very good AI feature, but it doesn't run more than two minutes.” So we have to balance the act of creating applications that are actually tailored toward the experience, and then some of the time, a lot of time actually, battery efficiency is a very important parameter. So that's why it kicks into this kind of different angle. I always think the NPU may have less of the pure raw throughput. It will not be pumping as many things, but it's so good at crunching things. Think about those small motor heat pumps. They do a lot of work efficiently. And then you can think that the GPU can be the furnace. Really put the gas in and it will get the job done quickly. And then the CPU will be just the robot that cleans my house, it'll do everything. So that's how I always see this kind of approach for this one. And then in particular, what's updated for this GPU is the performance actually a lot. The jump is huge. Forget about just AI for a second. I’ll use gaming for an example. I actually can play games now, actually proper 1080p games. So what that means is that there's raw performance in that. It's actually simple math. Think about how the GPU can do X amount of computation. They're doing graphics, and that's how CUDA started in the old days, it was fragment programs. By the way, I did a PhD on that back in the old days. I did CUDA code for a moment, I was like, “Okay, whatever that means,” but I was programming down there. I was like, “Okay, what it means is the number of threads you can do.” Same idea here. These new Intel Arc processors allow you to do more multithreaded tasks. They have more core in it, again. But I think they call it differently, they call it an execution unit. I think that's how Intel framed it. But what it means to you is that if you can do gaming faster by 2-3x, what do I expect? If you're not bandwidth limited to something memory related, the more performance wide, they are higher. So what I really like about this is, if I think about it, this is the worst laptop you can get today, by definition, and it's the best in the market. Worst in terms of what we're shipping is already done, but we have more coming up in the future. In a couple years, it's extremely unlikely we’ll ship something slower than this. That's what I mean by the worst and the best. It's the best in the market, but if you're a developer and you're developing for the future, we will not have a slower machine than this. What that means is that we create a baseline for the market right now to say, “Okay, I'm a developer, Here's what I can see, and I can expect this will give me enough performance maybe in one generation. Even this application doesn't support it. Then it will be great.” I can do, for example, Stable Diffusion that we talk about. It takes 5 seconds now instead of 10, it's a better experience. But I can still ship a 10 seconds one because it works, the software is there. I can still get the customer to try it because I don't think the person will complain too much about the 10 seconds, for example, because they get the work done on a battery that's small and also is portable. So this kind of future that we're seeing is quite exciting, and that's the balancing act that I think people can start looking into.
RD A lot of trade-offs.
BP To your point, it is super interesting this cross pollination between gaming and image work. I remember now thinking back, the company Movidius, they had spent a long time from 2005 to when I wrote about them trying to find product market fit and their stuff was created to help convert old movies into 3D to make more content for 3D TVs. They weren't thinking about AI or vision processing at all, but it was a happy accident and now it has the capability of doing a lot of those things.
RL Oh, that's cool. When I studied history I found it fascinating how hardware and software work together, because you can always have the best hardware, but no one uses it. It just has to have the balancing of what the needs are in the market at the right time. So we have a lot of skew of this. So this is the client version that I'm talking about, which is consumer laptops, desktops maybe, but we also have edge use cases that the variants of this can be used for. So it’s eventually actually filling the world with the silicon that can do the task, and then the form factor can be very different. I think that's one of the reasons why I joined Intel. My boss was telling me, “Have you tried to put something out there that millions of people already have?” I was like, “What do you mean?” x86 architecture? Okay, that's good. What about silicon, the Intel silicon side?” “Yeah, I think I heard about that when I was young. That's right.” So having that ecosystem that can help fuel this, because you know that the moment you finish building, there's a chance of it running at all like people trying to build cell phone apps back in the old days. Again, I use the phone because it's relevant to some of us. When I first built an iPhone app, for example, or Android app, you had to jailbreak almost to get an app on. I remember back in the old days you had to break the system to build a system. So that's the very, very much better outcome I see. If we all focus here, we can benefit a lot of people with not a lot of money, by the way. This machine is very cheap relative to everything else I've seen, the silicone and the package together and the capability.
RD So you're obviously the evangelist for OpenVINO and you have the hardware company that makes the hardware. I imagine the software is being optimized for the hardware pretty heavily. Does it ever go the other way? Is the hardware ever optimized for the software?
RL Very good question. I noticed that often it's a very difficult problem. For example, it requires years to change the architecture. So I think the best example is when we look at gaming and all that. I think back in the old days when the GPUs were developed, there was no need to use this for general purpose computing. There was not. Back then, the only solo task for the GPU back then was to make graphics, that's why it's called Graphics Processing Unit. And there's a momentum all of a sudden. People hacked it, like me, I was one of them, to make it general purpose. Then all of a sudden, the hardware people listening are like, “Oh my god, you guys are breaking what I designed it to do. You're not supposed to do that. You're not supposed to make the graphics go out, not to the display, but go back to the memory.” They worked backward, basically. They're just like, “The graphics memory should go this way,” but I was like, “No, I want to get the data back.” So we were hijacking the system at one point, and then we actually modified it.
RD It's just massively parallel floating point operations, right?
RL Yes, but it never existed before. The hardware didn't exist back then. Same here, the same story repeats. We will see some bandwidth limitations. There's certain criteria, let's say the compute is not efficient enough. Maybe we want to focus more on this, or maybe we want more threads or whatever. There's some design decision that will get inference based on the model that we're trying to optimize for. And then, I see this coming especially for large language models that are very bandwidth limited. Very likely, I may see in the future that the process of design may be more optimized toward how memory usage will look, for example. I'm just looking at the system itself as an engineer. And then I may also think about maybe all those low position support better. I don't know if anyone knows much about the compression for large language models. You never heard people doing floating point for that or inference. It just doesn't make sense, just too much compute. So these are the decisions we can make.
BP Ryan, you can probably speak to this with a more educated view than I can, but we actually have written about quantization and moving from floating point and how this stuff will be essential for getting this kind of powerful AI on device. And as you pointed out, that has multiple important uses. Maybe you don't want to burn as much battery, maybe you don't have access to great Wi-Fi. Maybe you want to keep things private, like you said. It's your device, why should it have to go to the cloud? And maybe for the company, that's better because they don't have to take your PII, which creates more overhead for them.
RL And actually that's what the inference looked like, to answer your question, that was designed for convolutional networks, but that's the path we actually created that silicon component that accelerates that part. And then we have XMX, AMX, all this computational acceleration also based on the model, because we do a lot of matrix multiplication and then now we accelerate those. So these are longer threads that we see happening, and in the shorter term, I think a lot of times it comes to software first, because it's a real story. Now people say I'm the expert. How did I start it? I was making really crappy code and I was reading Intel optimization examples to make it better. So everyone would start with the very non-optimal just to make it published. When I was doing PhD, I was like, “Make it five times faster.” I spent neverending nights to make it faster to make it publish, but that work that we started as a researcher often is with very limited resources and limited time we had to make it out. So a real example is, if you look at Stable Diffusion let's say one year from today, the quality is so much better. It’s like, “Is it real anymore? What's going on?” It's a lot better quality.
BP Oh, absolutely. There's all these Reddit threads and there was recently a New York Times quiz that most people cannot tell the difference between a deepfake and a real photo now if you create them.
RL That gave me even worries. We should be thinking more about what we're building, too. And then on the other side, the better quality at the same time comes with the burden of high compute and people find ways to reduce compute. So this triangle of ways how development usually works, there's no escape to that. Everyone wants something faster, better, and easier to do.
BP We see what's coming on the horizon. It's going to be text to video, and then it's going to be as close as possible real time text to video. And then as Ryan and I have discussed, it's going to be apps and services where the UI is adapted to you, and then it's going to be games where it's like, “What is your story today?” and the game will be generated on the fly. Really crazy stuff.
RL Oh, that would be so cool with mixed reality, too. Think about it– you can have a forest that's generated in front of your living room. That was one of my proposals back in the day when I was still at my old company. I was like, “I just want my own forest so I can be more calm in my own white room right now.” My room is plain white.
RD Just hallucinate on demand.
BP Whatever you need to feel good to get in the flow state, exactly.
[music plays]
BP All right, everybody. Thanks so much for listening. I hope you learned something today, and if you want to know more about what Intel is doing, we're going to let Raymond give his part of the outro and we'll definitely put a lot of links in the show notes. But before we get to that, as always, I want to thank someone who came on Stack Overflow and helped to spread a little knowledge. Awarded two days ago to Andrey Korneyev, “How to delete specific nodes from an XElement.” Andrey supplied the answer and was given a Lifeboat Badge for helping folks out. Over 30,000 people have viewed this question, so we appreciate it, Andrey. I am Ben Popper. I'm the Director of Content here at Stack Overflow. You can always find me on X @BenPopper or hit us up with an email– podcast@stackoverflow.com.
RD I'm Ryan Donovan. I edit the blog here at Stack Overflow. You can find it at stackoverflow.blog. And if you want to reach out to me on X, my handle is @RThorDonovan.
RL My name is Raymond Lo. You can find me on LinkedIn. You can find me on Twitter, just search for Raymond Lo Intel and most likely you'll see my link pop up. And more importantly, try all the demos I talked about today. I'm a developer, I want you to try what we started and make it a thing that you can make your life better with. For the demo, you can go to OpenVINO Notebooks. Look for that keyword, and you have all the Gen AI, chatbot, all the fun stuff you can run on your laptop. Enjoy.
BP Very cool. All right, everybody. Thanks for listening, and we will talk to you soon.
[outro music plays]