3 ms·

> Cuda would be pointless for Apple, because cuda is for Nvidia only, no? That is the point. Cuda is widely used in particular for training as opposed to tuni

> Cuda would be pointless for Apple, because cuda is for Nvidia only, no? That is the point. Cuda is widely used in particular for training as opposed to tuning and is just flat out not available. So you buy your nice $6000 machine and it just does not work for its intended purpose. > it’s the cheapest setup on the market that can get you this much VRAM. shared VRAM is not everything.

jeroenhd 3y ago
I think it's clear that GPU manufacturers are scamming their customers more than Apple does with their ridiculous prices. There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Nvidia intentionally nerfs the amount of VRAM in their consumer cards so you need to buy their ridiculously overpriced enterprise cards. I think it's fair to say Apple's lineup is the cheapest way to get >64GB of VRAM-ish memory and that's definitely not because Apple prices their products so fairly.
- hu3 3y ago
  > There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Those are not equivalent in speed. macs RAM are much slower than these GPUs.
  - jeroenhd 3y ago
    There is no Apple-to-apples comparison between a Macbook and a GPU. However, the problem with running most models is the lack of simultaneous RAM. You can swap memory back and forth between RAM and VRAM (with a huge performance penalty) of course, but that's not exactly usable or comparable to what Apple's VRAM sharing setup allows. Nvidia doesn't sell a nice-but-not-amazing GPU equivalent to Apple's processing power and memory bandwidth that's also capable of operating on >80GB of VRAM at once. Apple's SoC is kind of an oddball in that regard. I suppose you could take a regular old iGPU (for AMD, Intel, probably also Qualcom/Mediatek) and use its shared memory capabilities as a comparison. However, iGPUs are terrible at machine learning tasks, they don't come close to what Apple can do with their dedicated accelerators. The best middle ground may be the laptop GPUs with both dedicated RAM and shared RAM, but those will start swapping memory back and forth like crazy running large ML workloads so they're not really comparable. If you want to run a model that operates on a huge amount of memory at once, I don't think there is a desktop option that can do what Apple does without going for the massive overkill GPUs that will crush the Macbook in terms of performance (at great cost). Perhaps you know a GPU or iGPU that's capable of running 80GB VRAM workloads at comparable speeds? Because I don't.
    - diffeomorphism 3y ago
      That sounds like an overly specific target audience: - you need more than 16 or 32 GB of ram. - but less than 100. - speed is important. - but losing a factor 4 compared to a GPU is fine. - money is very limited. - but paying 6000 for a mac studio is cheap. - this is very important professional work. - but I don't need servers, ECC ram, raid, another OS... Don't get me wrong, I also think Nvidia is heavily price gouging, but there are definitely trade offs and it is not clear at all why 80GB is your magical number and not, say, 108 or 28.
idonotknowwhy 3y ago
Shared vram is everything of you're running local inference. My 3 x 3090s can't fit the 180b falcon even at 2 bits into vram. But people with a 128gb studio can, and it's cheaper

jeroenhd 3y ago

I think it's clear that GPU manufacturers are scamming their customers more than Apple does with their ridiculous prices. There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Nvidia intentionally nerfs the amount of VRAM in their consumer cards so you need to buy their ridiculously overpriced enterprise cards. I think it's fair to say Apple's lineup is the cheapest way to get >64GB of VRAM-ish memory and that's definitely not because Apple prices their products so fairly.

hu3 3y ago
> There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Those are not equivalent in speed. macs RAM are much slower than these GPUs.
- jeroenhd 3y ago
  There is no Apple-to-apples comparison between a Macbook and a GPU. However, the problem with running most models is the lack of simultaneous RAM. You can swap memory back and forth between RAM and VRAM (with a huge performance penalty) of course, but that's not exactly usable or comparable to what Apple's VRAM sharing setup allows. Nvidia doesn't sell a nice-but-not-amazing GPU equivalent to Apple's processing power and memory bandwidth that's also capable of operating on >80GB of VRAM at once. Apple's SoC is kind of an oddball in that regard. I suppose you could take a regular old iGPU (for AMD, Intel, probably also Qualcom/Mediatek) and use its shared memory capabilities as a comparison. However, iGPUs are terrible at machine learning tasks, they don't come close to what Apple can do with their dedicated accelerators. The best middle ground may be the laptop GPUs with both dedicated RAM and shared RAM, but those will start swapping memory back and forth like crazy running large ML workloads so they're not really comparable. If you want to run a model that operates on a huge amount of memory at once, I don't think there is a desktop option that can do what Apple does without going for the massive overkill GPUs that will crush the Macbook in terms of performance (at great cost). Perhaps you know a GPU or iGPU that's capable of running 80GB VRAM workloads at comparable speeds? Because I don't.
  - diffeomorphism 3y ago
    That sounds like an overly specific target audience: - you need more than 16 or 32 GB of ram. - but less than 100. - speed is important. - but losing a factor 4 compared to a GPU is fine. - money is very limited. - but paying 6000 for a mac studio is cheap. - this is very important professional work. - but I don't need servers, ECC ram, raid, another OS... Don't get me wrong, I also think Nvidia is heavily price gouging, but there are definitely trade offs and it is not clear at all why 80GB is your magical number and not, say, 108 or 28.

idonotknowwhy 3y ago

Shared vram is everything of you're running local inference. My 3 x 3090s can't fit the 180b falcon even at 2 bits into vram. But people with a 128gb studio can, and it's cheaper