4 ms·

One thing to keep in mind is that most of the "NPU" type on-chip accelerators are designed to minimize power consumption for small networks (think speech recogn

One thing to keep in mind is that most of the "NPU" type on-chip accelerators are designed to minimize power consumption for small networks (think speech recognition and video call background removal) rather than supporting big language models or anything you would do in say PyTorch. For those cases the path of least yak shaving is anything with an Ampere or newer NVIDIA GPU. I hold out some hope that the GPUs in AMD APUs will get better support but the hardware still isn't nearly as fast as contemporary NVIDIA chips.

dragontamer 2y ago
Its impossible for an iGPU to get any speed close to a dGPU, because dGPUs run independent RAM and independent cache (including gobs of "infinity" cache for AMD chips). Clocking memory at 500GB/s to 1000GB/s is a big deal when typical CPUs are only in the 50GB/s region (GPUs are on the order of 10x more bandwidth) ------- Building a latency-optimized kernel necessarily means you're programming against an obscure NPU / APU rather than the typical H100s. Performance is upside down compared to H100 or other typical GPUs, where CPU-to-GPU latency on an APU/NPU is negligible while memory-bandwidth is a precious resource. AMD made a bunch of kernels that did some silly video-based / camera-based gestures back in the Windows 8 days. So yeah, its possible to write efficient CPU-to-GPU integrated kernels that take advantage of zero-memory copies or other APU/NPU based tricks. Its just that if no popular APU/NPU comes out in the next decade, all that programming effort is for naught.
- hedgehog 2y ago
  The above poster is mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. Modern laptop chip memory bandwidths are in the 100GB/s to 400GB/s range and empirically a run of the mill Mac is enough for a lot of people to get started even with language models, the main issue being quality of software support. Edit: To be more precise current x86 laptop chips are more like 80GB/s - 120GB/s, have to get a Mac for more than that.
  - sva_ 2y ago
    I guess your post is rather charitable, and I'm a bit of a freetard (personally), but > mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. kind of triggered me a little. I'm an ML engineer and do not know any of my colleagues who run any serious training on their laptops, be it x86 or the latest Apple M-whatever. We all just use notebooks with GPUs hosted somewhere else. Perhaps the Apple M chips are a bit better at doing some inference, but at a certain level it is better to just use a dedicated GPU, and I think it will probably stay that way for a while (maybe forever). I have to say, the Apple M chips do impress me, and I wish there was some x86 equivalent, but alas there is none. And Linux support isn't anywhere close where I'd feel confident sacrificing my time sending out bug reports. I don't think any AMD NPU will replace NVIDIA GPUs anytime soon, and that wasn't the purpose of my post. I just wish AMD would actually put out a driver for me to be able to tinker with their hardware which so far they have not. And that sort of inability to even put out a driver to use their consumer-grade hardware does not inflict much confidence in their dedicated GPUs being a viable alternative to NVIDIA cards.
    - hedgehog 2y ago
      Hah, ok, yeah except for small org + tiny models no ML team is training on laptops. On the other hand I think we're already seeing more app devs using language models while not doing any training at all, and those people will probably want to work on their laptops. Think code completion, OCR / data extraction, image processing, or even as a local stand-in for cloud APIs similar to how SQLite is used as a database stand-in. In that sense local inference is pretty important.
- sliken 2y ago
  AMD/Intel CPUs for desktops/laptops are mostly 128 bit wide. So these days on laptops (no DIMM slots) is DDR5-6400 = 102GB/sec peak to DDR5-8000 (128GB/sec peak). Apple however has 192 bit (m3 pro), 256 bit (m2 pro), 512 bit (m3 max), and 1024 bit (m2 ultra) wide for improved iGPU performance. Generally CPU unified memory systems spend more transistors on cache than GPUs, so you also get some advantages there. iGPUs also have other advantages like much higher max ram, lower latency from CPU -> video memory than the traditional CPU -> PCIe -> GPU -> vram. Depending on what you are sending the cache coherent nature of iGPUs means you can have a complicated data structure and hand it to the GPU with a pointer instead of the normal CPU memory -> serialize -> transfer over PCIe -> poll or tell the GPU it's there, etc. So sure a gGPU is faster at LLMs, but unless you spend $$$$ on GPUs you can get much more ram with an iGPU. So even llama 70B runs decent on a M3 max (prompt @ 19 tokens/sec and eval rate at 8.5 tokens/sec). So sure it's not as fast as a dGPU, but I'd rather run 70B at 8-19 tokens/sec then not be able to run it on a 4090. The widely rumored AMD Strix Halo will have a 256 bit wide memory interface, which should significantly help the iGPU, much like the Apple M2 Pro.
sva_ 2y ago
I mean, of course. There's a reason why an h100 costs around $30k, if you even have a chance to buy it. For whom it may concern, some recent article about progress of the Linux xdna driver: https://www.phoronix.com/news/AMD-XDNA-Ryzen-AI-Driver-Patch https://www.phoronix.com/news/AMD-XDNA-Ryzen-AI-Driver-Patch
- hedgehog 2y ago
  No need for that, you can get pretty far with a $500 desktop GPU added to your existing machine or a $2000 gaming laptop (for inference workloads).

dragontamer 2y ago

Its impossible for an iGPU to get any speed close to a dGPU, because dGPUs run independent RAM and independent cache (including gobs of "infinity" cache for AMD chips). Clocking memory at 500GB/s to 1000GB/s is a big deal when typical CPUs are only in the 50GB/s region (GPUs are on the order of 10x more bandwidth) ------- Building a latency-optimized kernel necessarily means you're programming against an obscure NPU / APU rather than the typical H100s. Performance is upside down compared to H100 or other typical GPUs, where CPU-to-GPU latency on an APU/NPU is negligible while memory-bandwidth is a precious resource. AMD made a bunch of kernels that did some silly video-based / camera-based gestures back in the Windows 8 days. So yeah, its possible to write efficient CPU-to-GPU integrated kernels that take advantage of zero-memory copies or other APU/NPU based tricks. Its just that if no popular APU/NPU comes out in the next decade, all that programming effort is for naught.

hedgehog 2y ago
The above poster is mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. Modern laptop chip memory bandwidths are in the 100GB/s to 400GB/s range and empirically a run of the mill Mac is enough for a lot of people to get started even with language models, the main issue being quality of software support. Edit: To be more precise current x86 laptop chips are more like 80GB/s - 120GB/s, have to get a Mac for more than that.
- sva_ 2y ago
  I guess your post is rather charitable, and I'm a bit of a freetard (personally), but > mostly happy with a 7840HS which tells us they're a laptop user and likely looking at workloads that would work on a laptop with spare time hobby levels of effort. kind of triggered me a little. I'm an ML engineer and do not know any of my colleagues who run any serious training on their laptops, be it x86 or the latest Apple M-whatever. We all just use notebooks with GPUs hosted somewhere else. Perhaps the Apple M chips are a bit better at doing some inference, but at a certain level it is better to just use a dedicated GPU, and I think it will probably stay that way for a while (maybe forever). I have to say, the Apple M chips do impress me, and I wish there was some x86 equivalent, but alas there is none. And Linux support isn't anywhere close where I'd feel confident sacrificing my time sending out bug reports. I don't think any AMD NPU will replace NVIDIA GPUs anytime soon, and that wasn't the purpose of my post. I just wish AMD would actually put out a driver for me to be able to tinker with their hardware which so far they have not. And that sort of inability to even put out a driver to use their consumer-grade hardware does not inflict much confidence in their dedicated GPUs being a viable alternative to NVIDIA cards.
  - hedgehog 2y ago
    Hah, ok, yeah except for small org + tiny models no ML team is training on laptops. On the other hand I think we're already seeing more app devs using language models while not doing any training at all, and those people will probably want to work on their laptops. Think code completion, OCR / data extraction, image processing, or even as a local stand-in for cloud APIs similar to how SQLite is used as a database stand-in. In that sense local inference is pretty important.
sliken 2y ago
AMD/Intel CPUs for desktops/laptops are mostly 128 bit wide. So these days on laptops (no DIMM slots) is DDR5-6400 = 102GB/sec peak to DDR5-8000 (128GB/sec peak). Apple however has 192 bit (m3 pro), 256 bit (m2 pro), 512 bit (m3 max), and 1024 bit (m2 ultra) wide for improved iGPU performance. Generally CPU unified memory systems spend more transistors on cache than GPUs, so you also get some advantages there. iGPUs also have other advantages like much higher max ram, lower latency from CPU -> video memory than the traditional CPU -> PCIe -> GPU -> vram. Depending on what you are sending the cache coherent nature of iGPUs means you can have a complicated data structure and hand it to the GPU with a pointer instead of the normal CPU memory -> serialize -> transfer over PCIe -> poll or tell the GPU it's there, etc. So sure a gGPU is faster at LLMs, but unless you spend $$$$ on GPUs you can get much more ram with an iGPU. So even llama 70B runs decent on a M3 max (prompt @ 19 tokens/sec and eval rate at 8.5 tokens/sec). So sure it's not as fast as a dGPU, but I'd rather run 70B at 8-19 tokens/sec then not be able to run it on a 4090. The widely rumored AMD Strix Halo will have a 256 bit wide memory interface, which should significantly help the iGPU, much like the Apple M2 Pro.

sva_ 2y ago

I mean, of course. There's a reason why an h100 costs around $30k, if you even have a chance to buy it. For whom it may concern, some recent article about progress of the Linux xdna driver: https://www.phoronix.com/news/AMD-XDNA-Ryzen-AI-Driver-Patch https://www.phoronix.com/news/AMD-XDNA-Ryzen-AI-Driver-Patch

hedgehog 2y ago
No need for that, you can get pretty far with a $500 desktop GPU added to your existing machine or a $2000 gaming laptop (for inference workloads).