4 ms·

Do these AI accelerators even support common implementations for LLM inference or stable diffusion models? Last time I checked, AMD's support was terrible.

by kkzz99 3y ago

diggan 3y ago
I'm guessing no, as these newly announced processors are the first with those accelerators. > AMD is also bringing the power of a dedicated AI neural processing unit (NPU) to desktop PC processors for the first time with the introduction of Ryzen™ AI
- wtallis 3y ago
  They're merely the first desktop processors from AMD with these NPUs. This is last year's laptop silicon repackaged for their desktop socket. They also re-branded the laptop version with new model numbers, because laptop OEMs basically demand new model numbers on an annual cadence. This year their marketing for these chips is heavily emphasizing the NPUs, because they actually have some tooling for them now, but they were basically dead weight when the silicon first shipped.
zamalek 3y ago
AMD's support is terrible, if you believe their documentation. ROCm/hip works just fine (well, with Torch - Tensorflow is shamelessly an NVIDIA shill) on many "unsupported" GPUs if you enable an override envar.
- NekkoDroid 3y ago
  To add a bit more context: I remember reading somewhere (may have been in the Phoronix forum) from an official AMD engeneer that "supported" means validated and actively tested in their pipeline, while "unsupported" generally just means "we don't do any or minimal testing on these, they should work and we don't explicitly prevent them from working but we don't guarantee anything" (at least when it comes to same die/gen cards). In the same post they also wrote that they are gonna look at integrating more of those "unsupported" cards into their suite. Honestly, I hope they change their wording for this to something like "validated", "supported" and "unsupported" with actual explenations what each of these mean (fully tested, works theoretically, does not work even theoretically) Edit: I actually found the post I was talking about https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1428866-amd-rocm-6-0-now-available-to-download-with-mi300-support-pytorch-fp8-more-ai#post1428924 https://www.phoronix.com/forums/forum/linux-graphics-x-org-d...
  - makomk 3y ago
    ROCm is also incredibly fragile and buggy at the best of times, so anything not actively tested by them stands a good chance of not working. Hell, I remember that a while back people were having problems with machine learning code giving garbage results on one of the few consumer GPUs that was officially listed as supported and AMD eventually replying to the bug report and declaring that actually, no they weren't going to support it and they'd remove it from the list ratther than try and fix the issue. I think this was back when newer consumer GPUs were genuinely unsupported as in the code simply wasn't there too. Integrated GPUs have also alwayu had a lot of problems. There's also questionable OS compatiblity. ROCm is Linux-based and has extremely limited and rather experimental Windows support. Their fancy new neural processing unit is Windows-only, tied in with a Microsoft framework, and they don't seem to have any kind of definite plan for supporting it elsewhere. So there's quite possibly no single OS where all the hardware that theoretically could be used for machine learning and AI in these chips actually has even vaguely functioning code that works on that OS to make it work that way.
    - CuriousCosmic 3y ago
      > Their fancy new neural processing unit is Windows-only, tied in with a Microsoft framework, and they don't seem to have any kind of definite plan for supporting it elsewhere. So there's quite possibly no single OS where all the hardware that theoretically could be used for machine learning and AI in these chips actually has even vaguely functioning code that works on that OS to make it work that way. What they did was embed their Vitis IP in their CPUs. Those IP modules have been available in ASIC accelerator cards from AMD or their Xilinx FPGA offerings for a while now and have Linux support. So they are releasing these APUs (which are primarily focused at a windows audience) with Windows support via ONNX. And then once there are chips in circulation, AMD can get Linux Vitis support for these chips set up with members of the Linux community being able to review and validate those patchsets.
- wmf 3y ago
  Does ROCm even work on XDNA? I don't think it does?
- jiggawatts 3y ago
  Tensorflow is open source on GitHub. What’s stopping AMD engineers from contributing with Pull Requests? A casual search through the open PRs shows nothing of interest being submitted by team red.
- antx 3y ago
  ah yes, the good old HSA_OVERRIDE_GFX_VERSION=10.3.0 switcheroo
- snvzz 3y ago
  AMD is not at the edge of bankruptcy anymore. They should be able to afford testers, doing both manual and automated testing on a wide range of cards. ROCm's "supported hardware" list is small to the point of concerning. More cards also means more testers outside of AMD itself.
caycep 3y ago
as long as PyTorch supports it, does it matter?
- antx 3y ago
  for as long as I'll need to recompile PyTorch for it to adequately support my card, yes, it matters.
  - weebull 3y ago
    The problem there is with PyTorch. They've got bugs in their AMD build flow that people keep pointing out to them, but because it's people with consumer cards which are "unsupported" (which just means AMD won't personally solve your problems with those cards, not that they aren't intended to work) they close the issues / PRs without fixing the bug. AMDs wording isn't helping either of course.

Do these AI accelerators even support common implementations for LLM inference or stable diffusion models? Last time I checked, AMD's support was terrible.

by kkzz99 3y ago

Do these AI accelerators even support common implementations for LLM inference or stable diffusion models? Last time I checked, AMD's support was terrible.

diggan 3y ago

I'm guessing no, as these newly announced processors are the first with those accelerators. > AMD is also bringing the power of a dedicated AI neural processing unit (NPU) to desktop PC processors for the first time with the introduction of Ryzen™ AI

wtallis 3y ago
They're merely the first desktop processors from AMD with these NPUs. This is last year's laptop silicon repackaged for their desktop socket. They also re-branded the laptop version with new model numbers, because laptop OEMs basically demand new model numbers on an annual cadence. This year their marketing for these chips is heavily emphasizing the NPUs, because they actually have some tooling for them now, but they were basically dead weight when the silicon first shipped.

zamalek 3y ago

AMD's support is terrible, if you believe their documentation. ROCm/hip works just fine (well, with Torch - Tensorflow is shamelessly an NVIDIA shill) on many "unsupported" GPUs if you enable an override envar.

NekkoDroid 3y ago
To add a bit more context: I remember reading somewhere (may have been in the Phoronix forum) from an official AMD engeneer that "supported" means validated and actively tested in their pipeline, while "unsupported" generally just means "we don't do any or minimal testing on these, they should work and we don't explicitly prevent them from working but we don't guarantee anything" (at least when it comes to same die/gen cards). In the same post they also wrote that they are gonna look at integrating more of those "unsupported" cards into their suite. Honestly, I hope they change their wording for this to something like "validated", "supported" and "unsupported" with actual explenations what each of these mean (fully tested, works theoretically, does not work even theoretically) Edit: I actually found the post I was talking about https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1428866-amd-rocm-6-0-now-available-to-download-with-mi300-support-pytorch-fp8-more-ai#post1428924 https://www.phoronix.com/forums/forum/linux-graphics-x-org-d...
- makomk 3y ago
  ROCm is also incredibly fragile and buggy at the best of times, so anything not actively tested by them stands a good chance of not working. Hell, I remember that a while back people were having problems with machine learning code giving garbage results on one of the few consumer GPUs that was officially listed as supported and AMD eventually replying to the bug report and declaring that actually, no they weren't going to support it and they'd remove it from the list ratther than try and fix the issue. I think this was back when newer consumer GPUs were genuinely unsupported as in the code simply wasn't there too. Integrated GPUs have also alwayu had a lot of problems. There's also questionable OS compatiblity. ROCm is Linux-based and has extremely limited and rather experimental Windows support. Their fancy new neural processing unit is Windows-only, tied in with a Microsoft framework, and they don't seem to have any kind of definite plan for supporting it elsewhere. So there's quite possibly no single OS where all the hardware that theoretically could be used for machine learning and AI in these chips actually has even vaguely functioning code that works on that OS to make it work that way.
  - CuriousCosmic 3y ago
    > Their fancy new neural processing unit is Windows-only, tied in with a Microsoft framework, and they don't seem to have any kind of definite plan for supporting it elsewhere. So there's quite possibly no single OS where all the hardware that theoretically could be used for machine learning and AI in these chips actually has even vaguely functioning code that works on that OS to make it work that way. What they did was embed their Vitis IP in their CPUs. Those IP modules have been available in ASIC accelerator cards from AMD or their Xilinx FPGA offerings for a while now and have Linux support. So they are releasing these APUs (which are primarily focused at a windows audience) with Windows support via ONNX. And then once there are chips in circulation, AMD can get Linux Vitis support for these chips set up with members of the Linux community being able to review and validate those patchsets.
wmf 3y ago
Does ROCm even work on XDNA? I don't think it does?
jiggawatts 3y ago
Tensorflow is open source on GitHub. What’s stopping AMD engineers from contributing with Pull Requests? A casual search through the open PRs shows nothing of interest being submitted by team red.
antx 3y ago
ah yes, the good old HSA_OVERRIDE_GFX_VERSION=10.3.0 switcheroo
snvzz 3y ago
AMD is not at the edge of bankruptcy anymore. They should be able to afford testers, doing both manual and automated testing on a wide range of cards. ROCm's "supported hardware" list is small to the point of concerning. More cards also means more testers outside of AMD itself.

caycep 3y ago

as long as PyTorch supports it, does it matter?

antx 3y ago
for as long as I'll need to recompile PyTorch for it to adequately support my card, yes, it matters.
- weebull 3y ago
  The problem there is with PyTorch. They've got bugs in their AMD build flow that people keep pointing out to them, but because it's people with consumer cards which are "unsupported" (which just means AMD won't personally solve your problems with those cards, not that they aren't intended to work) they close the issues / PRs without fixing the bug. AMDs wording isn't helping either of course.