5 ms·

I feel like these huge graphics cards with insane amounts of RAM are the moat that AI companies have been hoping for. We can't possibly hope to run the kinds o

I feel like these huge graphics cards with insane amounts of RAM are the moat that AI companies have been hoping for. We can't possibly hope to run the kinds of models that run on 192GB of VRAM at home.

dmbaggett 2y ago
For inference you could use a maxed-out Mac Ultra; the RAM is shared between the CPU and GPU.
- alecco 2y ago
  For single user (batch_size = 1), sure. But that is quite expensive in $/tok.
jsheard 2y ago
Apple will gladly sell you a GPU with 192GB of memory, but your wallet won't like it.
- kbenson 2y ago
  Won't Nvidia, and Intel, and Qualcomm, and Falanx (who make the ARM Mali GPUs from what I can see), and Imagination Technologies (PowerVR) do the same? They each make a GPU, and if you pay them enough money I have a hard time beleiving they won't figure out how to slap enough RAM on a board for one of their existing products and making whatever changes are required.
- nextaccountic 2y ago
  The US government is looking into heavily limit availability of high end GPUs from now on. And the biggest and most effective bottleneck for AI right now is VRAM So maybe Apple is happy to sell huge GPUs like that but the government will probably put it under export controls like A100 and H100 already is
  - rbanffy 2y ago
    Cue the PowerMac G4 TV ad. https://youtu.be/lb7EhYy-2RE https://youtu.be/lb7EhYy-2RE
- rbanffy 2y ago
  OTOH, it comes free with one of the finest Unix workstations ever made.
  - Rinzler89 2y ago
    Which Unix workstation?
    - coolspot 2y ago
      They are referring to MacOS being included with expensive Mac hardware.
      - rbanffy 2y ago
        How many desktop systems can have 192GB visible to the GPU? How many cost less than a Mac?
        Rinzler89 2y ago
        Just because it has a lot of GPU RAM doesn't mean it's actually useful for people doing ML work. How many companies use Macs for ML work instead of Nvidia and Cuda?
        rbanffy 2y ago
        It won’t be as fast as a high-end GPU like the MI300 series, but it’s enough to check whether the code works before running it on a high-end GPU-heavy machine and the large GPU-accessible RAM simplifies the code enormously, as you don’t have to partition and shuffle data between CPU and GPU.
        Rinzler89 2y ago
        Ok that's the theory but how many companies actually do that for their workflow. All the ML companies I saw use directly Cuda for prototyping to production and don't bother with Apple ML unless their target happens to be exclusively iPhones.
tonetegeatinst 2y ago
Ow contrarily I'd argue the opposite. GPU vram has gotten faster but the density isn't that good. 8gb used to be high end for the early 2000's yet now 16gb can't even run games that well, especially if its a studio that loves vram. Side note: as someone who has been into machine learning for over 10 years, let me tell ya us hobbyists and researchers hunger for compute and memory. VRAM isn't everything.....I am well aware but certain workflows really do benefit from heaps of vram like vfx and cad and CFD. I realize that the dream of upgradable GPUs where I can upgrade the different components just like you do on the computer. Computer is slow, then upgrade ram or storage or get a faster chip that uses the same socket. GPU could possibility see modularity with the processor the vram etc. Level1Tech has some great videos about how PCIe is the future...where we can connect systems together using raw PCI lanes, which is similar to how nvidia Blackwell servers communicate to other servers in the rack.
- immibis 2y ago
  Wasn't that just because of Nvidia's market segmentation?
phkahler 2y ago
>> We can't possibly hope to run the kinds of models that run on 192GB of VRAM at home. I'm looking to build a mini-ITX system with 256GB of RAM for my next build. DDR5 spec can support that in 2 modules, but nobody makes them yet. No need for a GPU, I'm looking to the AMD APUs which are getting into the 50TOPs range. But yes, RAM seems to be the limiting factor. I'm a little surprised the memory companies aren't pushing harder for consumers to have that capacity.
- auspiv 2y ago
  128GB DDR5 module - https://store.supermicro.com/us_en/supermicro-hynix-128gb-288-pin-ddr5-4800-pc5-38400-server-memory-mem-dr512mh-er48.html https://store.supermicro.com/us_en/supermicro-hynix-128gb-28... It is of course RDIMM, but you didn't specify what memory type you were looking at.

dmbaggett 2y ago

For inference you could use a maxed-out Mac Ultra; the RAM is shared between the CPU and GPU.

alecco 2y ago
For single user (batch_size = 1), sure. But that is quite expensive in $/tok.

jsheard 2y ago

Apple will gladly sell you a GPU with 192GB of memory, but your wallet won't like it.

kbenson 2y ago
Won't Nvidia, and Intel, and Qualcomm, and Falanx (who make the ARM Mali GPUs from what I can see), and Imagination Technologies (PowerVR) do the same? They each make a GPU, and if you pay them enough money I have a hard time beleiving they won't figure out how to slap enough RAM on a board for one of their existing products and making whatever changes are required.
nextaccountic 2y ago
The US government is looking into heavily limit availability of high end GPUs from now on. And the biggest and most effective bottleneck for AI right now is VRAM So maybe Apple is happy to sell huge GPUs like that but the government will probably put it under export controls like A100 and H100 already is
- rbanffy 2y ago
  Cue the PowerMac G4 TV ad. https://youtu.be/lb7EhYy-2RE https://youtu.be/lb7EhYy-2RE
rbanffy 2y ago
OTOH, it comes free with one of the finest Unix workstations ever made.
- Rinzler89 2y ago
  Which Unix workstation?
  - coolspot 2y ago
    They are referring to MacOS being included with expensive Mac hardware.
    - rbanffy 2y ago
      How many desktop systems can have 192GB visible to the GPU? How many cost less than a Mac?
      - Rinzler89 2y ago
        Just because it has a lot of GPU RAM doesn't mean it's actually useful for people doing ML work. How many companies use Macs for ML work instead of Nvidia and Cuda?
        rbanffy 2y ago
        It won’t be as fast as a high-end GPU like the MI300 series, but it’s enough to check whether the code works before running it on a high-end GPU-heavy machine and the large GPU-accessible RAM simplifies the code enormously, as you don’t have to partition and shuffle data between CPU and GPU.
        Rinzler89 2y ago
        Ok that's the theory but how many companies actually do that for their workflow. All the ML companies I saw use directly Cuda for prototyping to production and don't bother with Apple ML unless their target happens to be exclusively iPhones.

tonetegeatinst 2y ago

Ow contrarily I'd argue the opposite. GPU vram has gotten faster but the density isn't that good. 8gb used to be high end for the early 2000's yet now 16gb can't even run games that well, especially if its a studio that loves vram. Side note: as someone who has been into machine learning for over 10 years, let me tell ya us hobbyists and researchers hunger for compute and memory. VRAM isn't everything.....I am well aware but certain workflows really do benefit from heaps of vram like vfx and cad and CFD. I realize that the dream of upgradable GPUs where I can upgrade the different components just like you do on the computer. Computer is slow, then upgrade ram or storage or get a faster chip that uses the same socket. GPU could possibility see modularity with the processor the vram etc. Level1Tech has some great videos about how PCIe is the future...where we can connect systems together using raw PCI lanes, which is similar to how nvidia Blackwell servers communicate to other servers in the rack.

immibis 2y ago
Wasn't that just because of Nvidia's market segmentation?

phkahler 2y ago

>> We can't possibly hope to run the kinds of models that run on 192GB of VRAM at home. I'm looking to build a mini-ITX system with 256GB of RAM for my next build. DDR5 spec can support that in 2 modules, but nobody makes them yet. No need for a GPU, I'm looking to the AMD APUs which are getting into the 50TOPs range. But yes, RAM seems to be the limiting factor. I'm a little surprised the memory companies aren't pushing harder for consumers to have that capacity.

auspiv 2y ago
128GB DDR5 module - https://store.supermicro.com/us_en/supermicro-hynix-128gb-288-pin-ddr5-4800-pc5-38400-server-memory-mem-dr512mh-er48.html https://store.supermicro.com/us_en/supermicro-hynix-128gb-28... It is of course RDIMM, but you didn't specify what memory type you were looking at.