4 ms·

Can they access the full RAM? Afaik they get capped to a portion of total available RAM. But to your other point, very little of the current popular ML stack d

by dagmx 2y ago

Can they access the full RAM? Afaik they get capped to a portion of total available RAM. But to your other point, very little of the current popular ML stack does more than CUDA and MPS. Some will do rocm but I don’t know if the AMD iGPUs are guaranteed to support it? There’s not much for Intel GPUs.

hedgehog 2y ago
It depends on the API used, whether the data is in the region considered "GPU memory" or whether it's shared with the compute API from the app's memory space. Support is somewhat in flux and I haven't been following closely but if you're curious this is my bookmarked jumping of point (a PyTorch ticket about this): https://github.com/pytorch/pytorch/issues/107605 https://github.com/pytorch/pytorch/issues/107605
- slavik81 2y ago
  My understanding is that as of Linux 6.10, the driver will now dynamically allocate more memory for the iGPU [1]. The driver team apparently reused a strategy that had been developed for MI300A. I'm hoping that in combination with the gfx11-generic ISA introduced in LLVM 18, this will make it straightforward to enable compute applications on both Phoenix and Strix (even if they are not officially supported by ROCm). [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.10-rc4&id=eb853413d02c8d9b27942429b261a9eef228f005 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - dagmx 2y ago
    FWIW, It’s one of the patches linked in their linked issue. However the commits do have some caveats called out, as do the techniques they use to achieve the higher allocations.
  - lhl 2y ago
    One issue is that even if you are using GTT (dynamically allocated memory), this is still limited as a percentage of total RAM. Eg, currently on my 7940HS, I have 64GB of memory, 8GB of dedicated VRAM (GART), and then a limit of 28GB of GTT - there is an amdgpu.gttsize parameter to "Restrict the size of GTT domain in MiB for testing. The default is -1 (It’s VRAM size if 3GB < VRAM < 3/4 RAM, otherwise 3/4 RAM size)", but I'm not sure what the practical/effective limits are.
- dagmx 2y ago
  I appreciate the link. The GTT and SDMA tricks mentioned there don’t really increase the shared ram use imho. They just increase the virtual memory the GPU can address, but with several tradeoffs in terms of allocation and copy operations. As an aside, it just feels like a lot of hacks that AMD and Intel should have handled ages ago for their iGPUs instead of letting them languish.
guilamu 2y ago
Be careful, most bios will let you use only 1/4 of the total ram for the integrated GPU. Some - really bad - bios are even limiting to 2gb totally ignoring how much ram is available.
sillystuff 2y ago
> Some will do rocm but I don’t know if the AMD iGPUs are guaranteed to support it? If you only care about inference, llama.cpp supports Vulkan on any iGPU with Vulkan drivers. On my laptop with crap bios that does not allow changing any video ram settings, reserved "vram" is 2GB, but llama.cpp-vulkan can access 16GB of "vram" (half of physical ram). 16GB vram is sufficient to run any model that has even remotely practical execution speed on my bottom-of-the-line ryzen 3 3250U (Picasso/Raven 2); you can always offload some layers to CPU to run even larger. (on Debian stable) Vulkan support: apt install libvulkan1 mesa-vulkan-drivers vulkan-tools Build deps for llama.cpp: apt install libshaderc-dev glslang-dev libvulkan-dev Build llama.cpp with vulkan back-end: make clean (I added this, in case you previously built with a diff back-end) make LLAMA_VULKAN=1 If more than one GPU: When running, you have to set GGML_VK_VISIBLE_DEVICES to the indices of the devices you want e.g., export GGML_VK_VISIBLE_DEVICES=0,1,2 The indices correspond to the device order in vulkaninfo --summary. By default llama.cpp will only use the first device it finds. llama.cpp-vulkan has worked really well, for me. But, per benchmarks from back when Vulkan support was first released, using the CUDA back-end was faster than the Vulkan back-end on NVIDIA GPUs. Probably same Rocm vs Vulkan on AMD too. But, zero non-free / binary blobs required for Vulkan, and Vulkan supports more devices (e.g., my iGPU is not supported by Rocm)-- haven't tried, but you can probably mix GPUs from diff manufacturers using Vulkan.

by dagmx 2y ago

hedgehog 2y ago

It depends on the API used, whether the data is in the region considered "GPU memory" or whether it's shared with the compute API from the app's memory space. Support is somewhat in flux and I haven't been following closely but if you're curious this is my bookmarked jumping of point (a PyTorch ticket about this): https://github.com/pytorch/pytorch/issues/107605 https://github.com/pytorch/pytorch/issues/107605

slavik81 2y ago
My understanding is that as of Linux 6.10, the driver will now dynamically allocate more memory for the iGPU [1]. The driver team apparently reused a strategy that had been developed for MI300A. I'm hoping that in combination with the gfx11-generic ISA introduced in LLVM 18, this will make it straightforward to enable compute applications on both Phoenix and Strix (even if they are not officially supported by ROCm). [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v6.10-rc4&id=eb853413d02c8d9b27942429b261a9eef228f005 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- dagmx 2y ago
  FWIW, It’s one of the patches linked in their linked issue. However the commits do have some caveats called out, as do the techniques they use to achieve the higher allocations.
- lhl 2y ago
  One issue is that even if you are using GTT (dynamically allocated memory), this is still limited as a percentage of total RAM. Eg, currently on my 7940HS, I have 64GB of memory, 8GB of dedicated VRAM (GART), and then a limit of 28GB of GTT - there is an amdgpu.gttsize parameter to "Restrict the size of GTT domain in MiB for testing. The default is -1 (It’s VRAM size if 3GB < VRAM < 3/4 RAM, otherwise 3/4 RAM size)", but I'm not sure what the practical/effective limits are.
dagmx 2y ago
I appreciate the link. The GTT and SDMA tricks mentioned there don’t really increase the shared ram use imho. They just increase the virtual memory the GPU can address, but with several tradeoffs in terms of allocation and copy operations. As an aside, it just feels like a lot of hacks that AMD and Intel should have handled ages ago for their iGPUs instead of letting them languish.

guilamu 2y ago

Be careful, most bios will let you use only 1/4 of the total ram for the integrated GPU. Some - really bad - bios are even limiting to 2gb totally ignoring how much ram is available.

sillystuff 2y ago

> Some will do rocm but I don’t know if the AMD iGPUs are guaranteed to support it? If you only care about inference, llama.cpp supports Vulkan on any iGPU with Vulkan drivers. On my laptop with crap bios that does not allow changing any video ram settings, reserved "vram" is 2GB, but llama.cpp-vulkan can access 16GB of "vram" (half of physical ram). 16GB vram is sufficient to run any model that has even remotely practical execution speed on my bottom-of-the-line ryzen 3 3250U (Picasso/Raven 2); you can always offload some layers to CPU to run even larger. (on Debian stable) Vulkan support: apt install libvulkan1 mesa-vulkan-drivers vulkan-tools Build deps for llama.cpp: apt install libshaderc-dev glslang-dev libvulkan-dev Build llama.cpp with vulkan back-end: make clean (I added this, in case you previously built with a diff back-end) make LLAMA_VULKAN=1 If more than one GPU: When running, you have to set GGML_VK_VISIBLE_DEVICES to the indices of the devices you want e.g., export GGML_VK_VISIBLE_DEVICES=0,1,2 The indices correspond to the device order in vulkaninfo --summary. By default llama.cpp will only use the first device it finds. llama.cpp-vulkan has worked really well, for me. But, per benchmarks from back when Vulkan support was first released, using the CUDA back-end was faster than the Vulkan back-end on NVIDIA GPUs. Probably same Rocm vs Vulkan on AMD too. But, zero non-free / binary blobs required for Vulkan, and Vulkan supports more devices (e.g., my iGPU is not supported by Rocm)-- haven't tried, but you can probably mix GPUs from diff manufacturers using Vulkan.