4 ms·

> For example, the latest MacBook Pro can have more than 60G+ unified GPU RAM that can be used to store the model weights and a reasonably powerful GPU to run m

> For example, the latest MacBook Pro can have more than 60G+ unified GPU RAM that can be used to store the model weights and a reasonably powerful GPU to run many workloads. ...for $3.5K minimum, according to the Apple website :/ Is there any chance WebGPU could utilize the matrix instructions shipping on newer/future IGPs? I think MLIR can do this through Vulkan, which is how SHARK is so fast in Stable Diffusion on the AMD 7900 series, but I know nothing about webgpu's restrictions or Apache TVM.

summarity 4y ago
That would need support in dawn: https://dawn.googlesource.com/dawn https://dawn.googlesource.com/dawn Dawn and WebIDL is also an easy way to add GPU support to any application (that can link C code (or use via a lib)). And Google maintains the compiler layer for the GPU frameworks (Metal, DX, Vulkan ...). This is going to be a great leap forward for GPGPU for many apps.
- brucethemoose2 4y ago
  Hmmm is there an issue tracker for Dawn matrix acceleration in... Vulkan or D3D12, I guess? Diving into these software stacks is making my head hurt.
satvikpendem 4y ago
Getting 64 GB of VRAM for $3.5k is a lot cheaper than buying the equivalent Nvidia discrete GPUs.
- summarity 4y ago
  It's also interesting that this opens up the full saturation of Apple Silicon (minus the ANE): GGML can run on the CPU, using NEON and AMX, while another instance could run via Metal on the GPU using MLC/dawn. Though the two couldn't share (the same) memory at the moment.
  - brucethemoose2 4y ago
    The GPU's ML task energy is so much lower that you'd probably get better performance running everything on the GPU. I think some repos have tried splitting things up between the NPU and GPU as well, but they didn't get good performance out of that combination? Not sure why, as the NPU is very low power.
  - j45 4y ago
    This was a really insightful explanation, thanks. I have been wanting to get a beefier Mac Studio/mini m2 the more I’m seeing Apple Silicon specific tweaked packages.
- verdverm 4y ago
  You can get it for a lot less from https://frame.work https://frame.work But 64G of VRAM is not the same as GPU mem, apples and oranges
  - satvikpendem 4y ago
    Where does Framework offer 64 GB of VRAM? By VRAM I am referring to GPU RAM, yes.
    - brucethemoose2 4y ago
      Technically any newish laptop with 64GB of RAM has 64GB of "VRAM," but right now the Apple M series and AMD 7000 series are the only IGPs with any significant ML power.
      - sroussey 4y ago
        I’m not sure what you mean. Typically, an iGPU slices off part of RAM for the GPU at boot time, which means it’s fixed and not shared. When did this change?
        brucethemoose2 4y ago
        Its fixed at boot but (on newer IGPs) can grow beyond the initial capacity.
        lights0123 4y ago
        For Intel, it seems that per their chart under "What is the maximum amount of graphics memory or video memory my computer can use?" and discussion under "Will adding more physical memory increase my graphics memory amount?" at https://www.intel.com/content/www/us/en/support/articles/000020962/graphics.html https://www.intel.com/content/www/us/en/support/articles/000..., iGPUs included with 5th gen/Broadwell processors were their first to do so in 2014.
        fulafel 4y ago
        Full unified memory came 10 ish years ago (also powering the PS4) but I think hw ability to adjust iGPU memory without booting predated that, Intel seems to have called it DVMT.
- brucethemoose2 4y ago
  Maybe Intel will start offering cheap, high capacity ARC dGPUs as a power play? That would certainly be disruptive. But yeah, AMD/Nvidia are never going offer huge memory pools affordably on dGPUs.

summarity 4y ago

That would need support in dawn: https://dawn.googlesource.com/dawn https://dawn.googlesource.com/dawn Dawn and WebIDL is also an easy way to add GPU support to any application (that can link C code (or use via a lib)). And Google maintains the compiler layer for the GPU frameworks (Metal, DX, Vulkan ...). This is going to be a great leap forward for GPGPU for many apps.

brucethemoose2 4y ago
Hmmm is there an issue tracker for Dawn matrix acceleration in... Vulkan or D3D12, I guess? Diving into these software stacks is making my head hurt.

satvikpendem 4y ago

Getting 64 GB of VRAM for $3.5k is a lot cheaper than buying the equivalent Nvidia discrete GPUs.

summarity 4y ago
It's also interesting that this opens up the full saturation of Apple Silicon (minus the ANE): GGML can run on the CPU, using NEON and AMX, while another instance could run via Metal on the GPU using MLC/dawn. Though the two couldn't share (the same) memory at the moment.
- brucethemoose2 4y ago
  The GPU's ML task energy is so much lower that you'd probably get better performance running everything on the GPU. I think some repos have tried splitting things up between the NPU and GPU as well, but they didn't get good performance out of that combination? Not sure why, as the NPU is very low power.
- j45 4y ago
  This was a really insightful explanation, thanks. I have been wanting to get a beefier Mac Studio/mini m2 the more I’m seeing Apple Silicon specific tweaked packages.
verdverm 4y ago
You can get it for a lot less from https://frame.work https://frame.work But 64G of VRAM is not the same as GPU mem, apples and oranges
- satvikpendem 4y ago
  Where does Framework offer 64 GB of VRAM? By VRAM I am referring to GPU RAM, yes.
  - brucethemoose2 4y ago
    Technically any newish laptop with 64GB of RAM has 64GB of "VRAM," but right now the Apple M series and AMD 7000 series are the only IGPs with any significant ML power.
    - sroussey 4y ago
      I’m not sure what you mean. Typically, an iGPU slices off part of RAM for the GPU at boot time, which means it’s fixed and not shared. When did this change?
      - brucethemoose2 4y ago
        Its fixed at boot but (on newer IGPs) can grow beyond the initial capacity.
      - lights0123 4y ago
        For Intel, it seems that per their chart under "What is the maximum amount of graphics memory or video memory my computer can use?" and discussion under "Will adding more physical memory increase my graphics memory amount?" at https://www.intel.com/content/www/us/en/support/articles/000020962/graphics.html https://www.intel.com/content/www/us/en/support/articles/000..., iGPUs included with 5th gen/Broadwell processors were their first to do so in 2014.
      - fulafel 4y ago
        Full unified memory came 10 ish years ago (also powering the PS4) but I think hw ability to adjust iGPU memory without booting predated that, Intel seems to have called it DVMT.
brucethemoose2 4y ago
Maybe Intel will start offering cheap, high capacity ARC dGPUs as a power play? That would certainly be disruptive. But yeah, AMD/Nvidia are never going offer huge memory pools affordably on dGPUs.