4 ms·

Still waiting for the Zen4 xdna NPU to be supported under Linux. Crossing my fingers that it'll actually be in 6.10 stable release. I'm generally happy with th

by sva_ 2y ago

Still waiting for the Zen4 xdna NPU to be supported under Linux. Crossing my fingers that it'll actually be in 6.10 stable release. I'm generally happy with the 7840HS, but didn't previously consider that large sections of the die are simply unusable under Linux for lack of a driver.

hedgehog 2y ago
I don't know about the NPU but 6.10 fixes one of the big memory issues that affected deep learning workloads on APUs.
dragontamer 2y ago
Is there any reason to wait for the NPU when GPUs are already available? I have to imagine that today's code should assume a long delay / latency between CPU and GPU. Yeah, NPUs allow CPU/GPU communication to be sped up, but are we in a position where we should be relying upon fast CPU/GPU communications yet?
- sva_ 2y ago
  I prefer not to have a dedicated GPU in my everyday laptop. I'd find it cool if AMD took their Linux userbase more seriously (there are dozens of us!!1) and actually provided drivers to use the hardware they sell.
  - hedgehog 2y ago
    One thing to keep in mind is that most of the "NPU" type on-chip accelerators are designed to minimize power consumption for small networks (think speech recognition and video call background removal) rather than supporting big language models or anything you would do in say PyTorch. For those cases the path of least yak shaving is anything with an Ampere or newer NVIDIA GPU. I hold out some hope that the GPUs in AMD APUs will get better support but the hardware still isn't nearly as fast as contemporary NVIDIA chips.
    - dragontamer 2y ago
      Its impossible for an iGPU to get any speed close to a dGPU, because dGPUs run independent RAM and independent cache (including gobs of "infinity" cache for AMD chips). Clocking memory at 500GB/s to 1000GB/s is a big deal when typical CPUs are only in the 50GB/s region (GPUs are on the order of 10x more bandwidth) ------- Building a latency-optimized kernel necessarily means you're programming against an obscure NPU / APU rather than the typical H100s. Performance is upside down compared to H100 or other typical GPUs, where CPU-to-GPU latency on an APU/NPU is negligible while memory-bandwidth is a precious resource. AMD made a bunch of kernels that did some silly video-based / camera-based gestures back in the Windows 8 days. So yeah, its possible to write efficient CPU-to-GPU integrated kernels that take advantage of zero-memory copies or other APU/NPU based tricks. Its just that if no popular APU/NPU comes out in the next decade, all that programming effort is for naught.
      - hedgehog 2y ago
        The above poster is mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. Modern laptop chip memory bandwidths are in the 100GB/s to 400GB/s range and empirically a run of the mill Mac is enough for a lot of people to get started even with language models, the main issue being quality of software support. Edit: To be more precise current x86 laptop chips are more like 80GB/s - 120GB/s, have to get a Mac for more than that.
        sva_ 2y ago
        I guess your post is rather charitable, and I'm a bit of a freetard (personally), but > mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. kind of triggered me a little. I'm an ML engineer and do not know any of my colleagues who run any serious training on their laptops, be it x86 or the latest Apple M-whatever. We all just use notebooks with GPUs hosted somewhere else. Perhaps the Apple M chips are a bit better at doing some inference, but at a certain level it is better to just use a dedicated GPU, and I think it will probably stay that way for a while (maybe forever). I have to say, the Apple M chips do impress me, and I wish there was some x86 equivalent, but alas there is none. And Linux support isn't anywhere close where I'd feel confident sacrificing my time sending out bug reports. I don't think any AMD NPU will replace NVIDIA GPUs anytime soon, and that wasn't the purpose of my post. I just wish AMD would actually put out a driver for me to be able to tinker with their hardware which so far they have not. And that sort of inability to even put out a driver to use their consumer-grade hardware does not inflict much confidence in their dedicated GPUs being a viable alternative to NVIDIA cards.
tracker1 2y ago
Just setup ChimeraOS on a 7945HS box for HTPC/Gaming chores... seems relatively nice so far, but still wanting to get a few things running on it.

by sva_ 2y ago

hedgehog 2y ago

I don't know about the NPU but 6.10 fixes one of the big memory issues that affected deep learning workloads on APUs.

dragontamer 2y ago

Is there any reason to wait for the NPU when GPUs are already available? I have to imagine that today's code should assume a long delay / latency between CPU and GPU. Yeah, NPUs allow CPU/GPU communication to be sped up, but are we in a position where we should be relying upon fast CPU/GPU communications yet?

sva_ 2y ago
I prefer not to have a dedicated GPU in my everyday laptop. I'd find it cool if AMD took their Linux userbase more seriously (there are dozens of us!!1) and actually provided drivers to use the hardware they sell.
- hedgehog 2y ago
  One thing to keep in mind is that most of the "NPU" type on-chip accelerators are designed to minimize power consumption for small networks (think speech recognition and video call background removal) rather than supporting big language models or anything you would do in say PyTorch. For those cases the path of least yak shaving is anything with an Ampere or newer NVIDIA GPU. I hold out some hope that the GPUs in AMD APUs will get better support but the hardware still isn't nearly as fast as contemporary NVIDIA chips.
  - dragontamer 2y ago
    Its impossible for an iGPU to get any speed close to a dGPU, because dGPUs run independent RAM and independent cache (including gobs of "infinity" cache for AMD chips). Clocking memory at 500GB/s to 1000GB/s is a big deal when typical CPUs are only in the 50GB/s region (GPUs are on the order of 10x more bandwidth) ------- Building a latency-optimized kernel necessarily means you're programming against an obscure NPU / APU rather than the typical H100s. Performance is upside down compared to H100 or other typical GPUs, where CPU-to-GPU latency on an APU/NPU is negligible while memory-bandwidth is a precious resource. AMD made a bunch of kernels that did some silly video-based / camera-based gestures back in the Windows 8 days. So yeah, its possible to write efficient CPU-to-GPU integrated kernels that take advantage of zero-memory copies or other APU/NPU based tricks. Its just that if no popular APU/NPU comes out in the next decade, all that programming effort is for naught.
    - hedgehog 2y ago
      The above poster is mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. Modern laptop chip memory bandwidths are in the 100GB/s to 400GB/s range and empirically a run of the mill Mac is enough for a lot of people to get started even with language models, the main issue being quality of software support. Edit: To be more precise current x86 laptop chips are more like 80GB/s - 120GB/s, have to get a Mac for more than that.
      - sva_ 2y ago
        I guess your post is rather charitable, and I'm a bit of a freetard (personally), but > mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. kind of triggered me a little. I'm an ML engineer and do not know any of my colleagues who run any serious training on their laptops, be it x86 or the latest Apple M-whatever. We all just use notebooks with GPUs hosted somewhere else. Perhaps the Apple M chips are a bit better at doing some inference, but at a certain level it is better to just use a dedicated GPU, and I think it will probably stay that way for a while (maybe forever). I have to say, the Apple M chips do impress me, and I wish there was some x86 equivalent, but alas there is none. And Linux support isn't anywhere close where I'd feel confident sacrificing my time sending out bug reports. I don't think any AMD NPU will replace NVIDIA GPUs anytime soon, and that wasn't the purpose of my post. I just wish AMD would actually put out a driver for me to be able to tinker with their hardware which so far they have not. And that sort of inability to even put out a driver to use their consumer-grade hardware does not inflict much confidence in their dedicated GPUs being a viable alternative to NVIDIA cards.

tracker1 2y ago

Just setup ChimeraOS on a 7945HS box for HTPC/Gaming chores... seems relatively nice so far, but still wanting to get a few things running on it.