4 ms·

Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.

slaymaker1907 4y ago
Nvidia actually does support this via unified memory. It’s actually an amazing performance trick since you can avoid launching a bunch of kernels just to ferry data back and forth. I did this on a a GTX 1080 for a school project to speed up random shuffling of large amounts of data. However, even without this feature, you can implement this sort of thing manually in most cases and you’re already being careful on a GPU to respect the cache (only working with one contiguous set of data of memory at a time). Really, we just need some good systems devs working on running these huge models.
PragmaticPulp 4y ago
PCIe devices like GPUs can access system memory. Integrated GPUs also access system memory via the same bus as the CPU. It’s not really a new technique. Apple just shipped a highly integrated unit with large memory bandwidth.
- recuter 4y ago
  That's a pretty important just. They chose to go down this path at this time and shipped, now I can run the 64B Llama on a widely available $3,999 prosumer device. How much would a PC that can do that currently cost me and can I have it by tomorrow?
  - smoldesu 4y ago
    puts on my jaded realist hat It's a great option if you have the hardware and want the speed. It's table stakes when other vendors like Nvidia, Intel, Qualcomm and Microsoft have acceleration though. Raw-compute-for-the-buck has always been a blowout with Apple Silicon GPUs, and it's not any prettier now that 4nm 40XX series cards are available. Hell, an Intel A770 with 16gb of VRAM is still cheaper than adding 16gb of RAM to a Mac Mini. It's good stuff, but pitched a bit hard with all the marketing. From where I'm standing, it looks like Apple is playing catch-up with their GPUs and acceleration tech. tentatively removes hat of jaded realism
  - jokethrowaway 4y ago
    You can buy a prebuilt pc with a 4090 for less - which is significantly more powerful but still in the 3xxx$. You could go cheaper with a 3090 which has the same vram and it's just slower. I think the best combo is a serious Nvidia pc for AI + a cheap MacBook air for portability.
    - jokethrowaway 4y ago
      There's even an Asus laptop with a 4099 for 3999$
      - fomine3 4y ago
        Laptop 4090 is desktop 4080
    - MacsHeadroom 4y ago
      You need two 3090s or 4090s to fit 65B even in 4bit. It's a big one. That said, if you're fine with slower speeds then two P40s could get the job done for $150 each. (Not sure how much slower this would go though.)
  - KVFinn 4y ago
    >How much would a PC that can do that currently cost me and can I have it by tomorrow? At the moment, seems like Apple has an edge here. On PC for single GPU you need an NVIDIA A40, which used prices for is about $2500, and not at retail stores. If you don't mind having two GPUs then two $800 3090 GPUs works, but that's a workstation build you'll have to order from Puget or something. That's probably faster than Apple. My gut instinct is that there's some low hanging fruit here and in the next couple weeks 64B Llama will run comparably or faster on any PC with a single 4090/3090 and 64 or 128 GB of system memory. But probably not any PC laptops that aren't 17 inch beasts, Apple will keep that advantage.
    - brookst 4y ago
      …and for models that require 64GB of VRAM? 120GB of VRAM? You can get a 128GB UMA mac for less than a single 48GB a100, let alone a single 96GB a100. I think Apple got incredibly lucky here, but I don’t see how the PC world catches them any time soon. We’ve all known that UMA is theoretically better for ages, but Apple’s timing couldn’t be better. And scale economies mean they can sell the same chip to people who need 100GB of system RAM and people who need 100GB of VRAM. If they can get their GPU / neural performance up and sort out their terrible relationship with academic research, they could snipe ML away from nvidia. It seems very unlikely, but it’s kind of stunning that it’s even in the realm of possibility.
      - smoldesu 4y ago
        > they could snipe ML away from nvidia. If Nvidia announced tomorrow that they were cancelling every datacenter deal they had, open-sourcing CUDA and publishing their entire patent library to the creative commons, I would still not believe you. This is a fun project for people with Apple Silicon machines who want to participate in the AI happenings, but I don't think you can warp it into a call for Nvidia's head. Let's wait until Apple pulls the curtains on their rackmount Mac Pros, so we can compare it with Nvidia's ARM server offerings: https://www.nvidia.com/en-us/data-center/grace-cpu/ https://www.nvidia.com/en-us/data-center/grace-cpu/
        4y ago
kkielhofner 4y ago
The Nvidia Jetson/Tegra line did (and does) unified memory. Released 2014[0]. Nvidia already has ARM cores. They’re just terrible compared to Apple, Samsung, etc. This is what the attempted ARM acquisition by Nvidia was about - with the ARM talent, IP, etc they’d be able to integrate more than just memory (GPU, CPU, connectivity via Mellanox, etc). Regulators shut it down (for good reason) but I can’t help but think we would have seen some really interesting and revolutionary platforms come from it. [0] - https://en.m.wikipedia.org/wiki/Tegra#Tegra_K1 https://en.m.wikipedia.org/wiki/Tegra#Tegra_K1
userbinator 4y ago
Intel GPUs have had that feature for over two decades, and it was also called UMA; synonymous with cheap and slow, before Apple hyped that term and made a UMA system that actually had decent performance.
dagmx 4y ago
Both AMD and Intel have APUs. However they’re limited by being minority products (so very low support from software) and often have limits on how much ram can be accessed by the GPU and/or have very weak GPUs.
shaklee3 4y ago
as pointed out, both Jetson and UM do this, but the upcoming Grace does it at higher bandwidth than the apple chip.
fulafel 4y ago
They have, see eg this from 10 years ago: https://www.tomshardware.com/news/AMD-HSA-hUMA-APU,22324.html https://www.tomshardware.com/news/AMD-HSA-hUMA-APU,22324.htm... Implementing the software support and getting operating systems to play along and fragmentation between GPU vendors, as always with GPUs on x86, have been the problems. From all accounts it's been working reasonably well on the consoles though. Also chicken-and-egg: low GPU compute usage uptake outside of games has meant it's not improved lately.
mhh__ 4y ago
1. They have 2. CPUs and GPUs typically disagree on whether they want high bandwidth or low latency, apple managed to keep both happy but it's very hard to do on a PC where the RAM,CPU, and GPU are quite far apart and also nowhere near as homogenous as Apple have them.
chucklenorris 4y ago
Games don't want slow memory accesses because it tanks the fps so there was no incentive to have a working implementation (except in laptops). And also - vram is a moat that keeps the cost of "professional" cards absurdly high without making them actually faster than consumer cards.
imtringued 4y ago
Nvidia didn't implement it in their OpenCL drivers because OpenCL's shared virtual memory spec is so terrible that khronos made it optional in OpenCL 3.0 and Intel (the only company actually having high quality OpenCL implementations) dropped shared virtual memory like a hot potato and instead introduced their own unified shared memory extension for OpenCL which they use for SyCL and oneAPI.

slaymaker1907 4y ago

Nvidia actually does support this via unified memory. It’s actually an amazing performance trick since you can avoid launching a bunch of kernels just to ferry data back and forth. I did this on a a GTX 1080 for a school project to speed up random shuffling of large amounts of data. However, even without this feature, you can implement this sort of thing manually in most cases and you’re already being careful on a GPU to respect the cache (only working with one contiguous set of data of memory at a time). Really, we just need some good systems devs working on running these huge models.

PragmaticPulp 4y ago

PCIe devices like GPUs can access system memory. Integrated GPUs also access system memory via the same bus as the CPU. It’s not really a new technique. Apple just shipped a highly integrated unit with large memory bandwidth.

recuter 4y ago
That's a pretty important just. They chose to go down this path at this time and shipped, now I can run the 64B Llama on a widely available $3,999 prosumer device. How much would a PC that can do that currently cost me and can I have it by tomorrow?
- smoldesu 4y ago
  puts on my jaded realist hat It's a great option if you have the hardware and want the speed. It's table stakes when other vendors like Nvidia, Intel, Qualcomm and Microsoft have acceleration though. Raw-compute-for-the-buck has always been a blowout with Apple Silicon GPUs, and it's not any prettier now that 4nm 40XX series cards are available. Hell, an Intel A770 with 16gb of VRAM is still cheaper than adding 16gb of RAM to a Mac Mini. It's good stuff, but pitched a bit hard with all the marketing. From where I'm standing, it looks like Apple is playing catch-up with their GPUs and acceleration tech. tentatively removes hat of jaded realism
- jokethrowaway 4y ago
  You can buy a prebuilt pc with a 4090 for less - which is significantly more powerful but still in the 3xxx$. You could go cheaper with a 3090 which has the same vram and it's just slower. I think the best combo is a serious Nvidia pc for AI + a cheap MacBook air for portability.
  - jokethrowaway 4y ago
    There's even an Asus laptop with a 4099 for 3999$
    - fomine3 4y ago
      Laptop 4090 is desktop 4080
  - MacsHeadroom 4y ago
    You need two 3090s or 4090s to fit 65B even in 4bit. It's a big one. That said, if you're fine with slower speeds then two P40s could get the job done for $150 each. (Not sure how much slower this would go though.)
- KVFinn 4y ago
  >How much would a PC that can do that currently cost me and can I have it by tomorrow? At the moment, seems like Apple has an edge here. On PC for single GPU you need an NVIDIA A40, which used prices for is about $2500, and not at retail stores. If you don't mind having two GPUs then two $800 3090 GPUs works, but that's a workstation build you'll have to order from Puget or something. That's probably faster than Apple. My gut instinct is that there's some low hanging fruit here and in the next couple weeks 64B Llama will run comparably or faster on any PC with a single 4090/3090 and 64 or 128 GB of system memory. But probably not any PC laptops that aren't 17 inch beasts, Apple will keep that advantage.
  - brookst 4y ago
    …and for models that require 64GB of VRAM? 120GB of VRAM? You can get a 128GB UMA mac for less than a single 48GB a100, let alone a single 96GB a100. I think Apple got incredibly lucky here, but I don’t see how the PC world catches them any time soon. We’ve all known that UMA is theoretically better for ages, but Apple’s timing couldn’t be better. And scale economies mean they can sell the same chip to people who need 100GB of system RAM and people who need 100GB of VRAM. If they can get their GPU / neural performance up and sort out their terrible relationship with academic research, they could snipe ML away from nvidia. It seems very unlikely, but it’s kind of stunning that it’s even in the realm of possibility.
    - smoldesu 4y ago
      > they could snipe ML away from nvidia. If Nvidia announced tomorrow that they were cancelling every datacenter deal they had, open-sourcing CUDA and publishing their entire patent library to the creative commons, I would still not believe you. This is a fun project for people with Apple Silicon machines who want to participate in the AI happenings, but I don't think you can warp it into a call for Nvidia's head. Let's wait until Apple pulls the curtains on their rackmount Mac Pros, so we can compare it with Nvidia's ARM server offerings: https://www.nvidia.com/en-us/data-center/grace-cpu/ https://www.nvidia.com/en-us/data-center/grace-cpu/
      - 4y ago

kkielhofner 4y ago

The Nvidia Jetson/Tegra line did (and does) unified memory. Released 2014[0]. Nvidia already has ARM cores. They’re just terrible compared to Apple, Samsung, etc. This is what the attempted ARM acquisition by Nvidia was about - with the ARM talent, IP, etc they’d be able to integrate more than just memory (GPU, CPU, connectivity via Mellanox, etc). Regulators shut it down (for good reason) but I can’t help but think we would have seen some really interesting and revolutionary platforms come from it. [0] - https://en.m.wikipedia.org/wiki/Tegra#Tegra_K1 https://en.m.wikipedia.org/wiki/Tegra#Tegra_K1

userbinator 4y ago

Intel GPUs have had that feature for over two decades, and it was also called UMA; synonymous with cheap and slow, before Apple hyped that term and made a UMA system that actually had decent performance.

dagmx 4y ago

Both AMD and Intel have APUs. However they’re limited by being minority products (so very low support from software) and often have limits on how much ram can be accessed by the GPU and/or have very weak GPUs.

shaklee3 4y ago

as pointed out, both Jetson and UM do this, but the upcoming Grace does it at higher bandwidth than the apple chip.

fulafel 4y ago

They have, see eg this from 10 years ago: https://www.tomshardware.com/news/AMD-HSA-hUMA-APU,22324.html https://www.tomshardware.com/news/AMD-HSA-hUMA-APU,22324.htm... Implementing the software support and getting operating systems to play along and fragmentation between GPU vendors, as always with GPUs on x86, have been the problems. From all accounts it's been working reasonably well on the consoles though. Also chicken-and-egg: low GPU compute usage uptake outside of games has meant it's not improved lately.

mhh__ 4y ago

1. They have 2. CPUs and GPUs typically disagree on whether they want high bandwidth or low latency, apple managed to keep both happy but it's very hard to do on a PC where the RAM,CPU, and GPU are quite far apart and also nowhere near as homogenous as Apple have them.

chucklenorris 4y ago

Games don't want slow memory accesses because it tanks the fps so there was no incentive to have a working implementation (except in laptops). And also - vram is a moat that keeps the cost of "professional" cards absurdly high without making them actually faster than consumer cards.

imtringued 4y ago

Nvidia didn't implement it in their OpenCL drivers because OpenCL's shared virtual memory spec is so terrible that khronos made it optional in OpenCL 3.0 and Intel (the only company actually having high quality OpenCL implementations) dropped shared virtual memory like a hot potato and instead introduced their own unified shared memory extension for OpenCL which they use for SyCL and oneAPI.