3 ms·

A decked out Mac Studio is like $7k for far less GPU power. I find that highly unlikely.

inciampati 2y ago
But you get access to a very large amount of RAM for that price.
softfalcon 2y ago
Don't attack me, I'm not disagreeing with you that an nVidia GPU is far superior at that price point. I simply want to point out that these folks don't really care about that. They want a Mac for more reasons than "performance per watt/dollar" and if it's "good enough", they'll pay that Apple tax. Yes, yes, I know, it's frustrating and they could get better Linux + GPU goodness with an nVidia PC running Ubuntu/Arch/Debian, but macOS is painless for the average science AI/ML training person to set up and work with. There are also known enterprise OS management solutions that business folks will happily sign off on. Also, $7000 is chump change in the land of "can I get this AI/ML dev to just get to work on my GPT model I'm using to convince some VC's to give me $25-500 million?" tldr; they're gonna buy a Mac cause it's a Mac and they want a Mac and their business uses Mac's. No amount of "but my nVidia GPU = better" is ever going to convince them otherwise as long as there is a "sort of" reasonable price point inside Apple's ecosystem.
- brookst 2y ago
  What Linux setup do you recommend for 128GB of GPU memory?
TylerE 2y ago
A non-decked out Mac Studio is a hell of a machine for $1999. Do you also compare cars by looking at only the super expensive limited editions, with every single option box ticked? I'd also point out that said 3 year old $1999 Mac Studio that I'm typing this on already runs ML models usefully, maybe 40-50% of the old 3000-series Nvidia machine it replaces, while using literally less than 10% of the power and making a tiny tiny fraction of the noise. Oh, and it was cheaper. And not running Windows.
- bee_rider 2y ago
  They are talking about training models, though. Run is a bit ambiguous, is that also what you mean?
  - TylerE 2y ago
    No. For training the Macs do have some interesting advantages due to the unified memory. The GPU cores have access to all of system RAM (and also the system RAM is ridiculously fast - 400GB/sec when DDR4 is barely 30GB/sec, which has a lot of little fringe benefits of it's own, part of why the Studio feels like an even more powerful machine than it actually is. It's just super snappy and responsive, even under heavy load.) The largest consumer NVidia card has 22GB of useable RAM. The $1999 Mac has 32GB, and for $400 more you get 64GB. $3200 gets you 96GB, and more GPU cores. You can hit the system max of 192GB for $5500 on an Ultra, albeit it with the lessor GPU. Even the recently announced 6000-series AI-oriented NVidia cards max out at 48GB. My understanding is a that a lot of enthusiasts are using Macs for training because for certain things having more RAM is just enabling.
    - Der_Einzige 2y ago
      The huge amount of optimizations available on Nvidia and not available on Apple make the reduced VRAM worth it, because even the most bloated of foundation models will have some magical 0.1bit quantization technique be invented by a turbo-nerd which only works on Nvidia. I keep hearing this meme of Mac's being a big deal in LLM training, but I have seen zero evidence of it, and I am deeply immersed in the world of LLM training, including training from scratch. Stop trying to meme apple M chips as AI accelerators. I'll believe it when unsloth starts to support a single non-nvidia chip.
      - arvinsim 2y ago
        Agreed. Although inference is good enough on the Mac, there is no way I am training on them at all. It's just more efficient to offload training to cloud Nvidia GPUs
      - 2y ago
andrewmcwatters 2y ago
Not all of us who own small businesses are out here speccing AMD Ryzen 9s and RTX 4090s for workstations. You can't lug around a desktop workstation.

But you get access to a very large amount of RAM for that price.

Don't attack me, I'm not disagreeing with you that an nVidia GPU is far superior at that price point. I simply want to point out that these folks don't really care about that. They want a Mac for more reasons than "performance per watt/dollar" and if it's "good enough", they'll pay that Apple tax. Yes, yes, I know, it's frustrating and they could get better Linux + GPU goodness with an nVidia PC running Ubuntu/Arch/Debian, but macOS is painless for the average science AI/ML training person to set up and work with. There are also known enterprise OS management solutions that business folks will happily sign off on. Also, $7000 is chump change in the land of "can I get this AI/ML dev to just get to work on my GPT model I'm using to convince some VC's to give me $25-500 million?" tldr; they're gonna buy a Mac cause it's a Mac and they want a Mac and their business uses Mac's. No amount of "but my nVidia GPU = better" is ever going to convince them otherwise as long as there is a "sort of" reasonable price point inside Apple's ecosystem.

brookst 2y ago
What Linux setup do you recommend for 128GB of GPU memory?

TylerE 2y ago

A non-decked out Mac Studio is a hell of a machine for $1999. Do you also compare cars by looking at only the super expensive limited editions, with every single option box ticked? I'd also point out that said 3 year old $1999 Mac Studio that I'm typing this on already runs ML models usefully, maybe 40-50% of the old 3000-series Nvidia machine it replaces, while using literally less than 10% of the power and making a tiny tiny fraction of the noise. Oh, and it was cheaper. And not running Windows.

bee_rider 2y ago
They are talking about training models, though. Run is a bit ambiguous, is that also what you mean?
- TylerE 2y ago
  No. For training the Macs do have some interesting advantages due to the unified memory. The GPU cores have access to all of system RAM (and also the system RAM is ridiculously fast - 400GB/sec when DDR4 is barely 30GB/sec, which has a lot of little fringe benefits of it's own, part of why the Studio feels like an even more powerful machine than it actually is. It's just super snappy and responsive, even under heavy load.) The largest consumer NVidia card has 22GB of useable RAM. The $1999 Mac has 32GB, and for $400 more you get 64GB. $3200 gets you 96GB, and more GPU cores. You can hit the system max of 192GB for $5500 on an Ultra, albeit it with the lessor GPU. Even the recently announced 6000-series AI-oriented NVidia cards max out at 48GB. My understanding is a that a lot of enthusiasts are using Macs for training because for certain things having more RAM is just enabling.
  - Der_Einzige 2y ago
    The huge amount of optimizations available on Nvidia and not available on Apple make the reduced VRAM worth it, because even the most bloated of foundation models will have some magical 0.1bit quantization technique be invented by a turbo-nerd which only works on Nvidia. I keep hearing this meme of Mac's being a big deal in LLM training, but I have seen zero evidence of it, and I am deeply immersed in the world of LLM training, including training from scratch. Stop trying to meme apple M chips as AI accelerators. I'll believe it when unsloth starts to support a single non-nvidia chip.
    - arvinsim 2y ago
      Agreed. Although inference is good enough on the Mac, there is no way I am training on them at all. It's just more efficient to offload training to cloud Nvidia GPUs
    - 2y ago

andrewmcwatters 2y ago

Not all of us who own small businesses are out here speccing AMD Ryzen 9s and RTX 4090s for workstations. You can't lug around a desktop workstation.