3 ms·

> its almost criminal to be recommending dedicated GPUs with only 12GB of RAM. If you already own a PC, it makes a hell of a lot more sense to spend $900 on a

> its almost criminal to be recommending dedicated GPUs with only 12GB of RAM. If you already own a PC, it makes a hell of a lot more sense to spend $900 on a 3090 than it does to spec out a Mac Mini with 24gb of RAM. Plus, the Nvidia setup can scale to as many GPUs as you own which gives you options for upgrading that Apple wouldn't be caught dead offering. Oh, and native Linux support that doesn't suck balls is a plus. I haven't benchmarked a Mac since the M2 generation, but the figures I can find put the M4 Max's compute somewhere near the desktop 3060 Ti: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks

adam_arthur 2y ago
A Mac Mini with 24GB is ~$800 at the cheapest configuration. I can respect wanting to do a single part upgrade, but if you're using these LLMs for serious work, the price/perf for inferencing is far in favor of using Macs at the moment. You can easily use the MacMini as a hub for running the LLM while you do work on your main computer (and it won't eat up your system resources or turn your primary computer into a heater) I hope that more non-mac PCs come out optimized for high RAM SoC, I'm personally not a huge Apple fan but use them begrudgingly. Also your $900 quote is a used/refurbished GPU. I've had plenty of GPUs burn out on me in the old days, not sure how it is nowadays, but that's a lot to pay for a used part IMO
- fragmede 2y ago
  if you're doing serious work, performance is more important than getting a good price/perf ratio, and a pair of 3090s is gonna be faster. It depends on your budget, however as that configuration is a bit more expensive, however.
  - adam_arthur 2y ago
    Whether performance or cost is more important depends on your use case. Some tasks that an LLM can do very well may not need to be done often, or even particularly quickly (as in my case). e.g. LLM as one step of an ETL-style pipeline Latency of the response really only matters if that response is user facing and is being actively awaited by the user
seanmcdirmid 2y ago
> M4 Max's compute somewhere near the desktop 3060 Ti The only advantage is the M4 Max's ability to have way more VRAM than a 3060 Ti. You won't find many M4 Maxes with just 8 or 16 GB of RAM, and I don't think you can do much except use really small models with a 3060 Ti.
- talldayo 2y ago
  It's a bit of a moot point when CUDA will run 4 3060Tis in parallel, with further options for paging out to system memory. Since most models (particularly bigger/MOE ones) are sparsely decoded, you can get quite a lot of mileage out of multiple PCIe slots fed with enough bandwidth. There's no doubt in my mind that the PC is the better performer if raw power is your concern. It's far-and-away the better value if you don't need to buy new hardware and only need a GPU. $2,000 of Nvidia GPUs will buy you halfway to an enterprise cluster, $2,000 of Apple hardware will get you a laptop chip with HBM.
  - seanmcdirmid 2y ago
    You need a lot of space for that, cooling, and a good fuse that won't trip when you turn it on. I would totally just pay the money for an M4 Ultra MacStudio with 128 GB of RAM (or an M4 Max with 64 GB). It is a much cleaner setup, especially if you aren't interested in image generation (which the Macs are not good at yet). If I could spend $4k on a non-Apple turn key solution that I could reasonably manage in my house, I would totally consider it.
    - talldayo 2y ago
      Well, that's your call. If you're the sort of person that's willing to spend $2,000 on a M4 Ultra (which doesn't quite exist yet but we can pretend it does), then I honest to god do not understand why you'd refuse to spend that same money on a Jetson Orin with the same amount of memory in a smaller footprint with better performance and lower power consumption. Unless you're specifically speccing out a computer for mobile use, the price premium you spend on a Mac isn't for better software or faster hardware. If you can tolerate Linux or Windows, I don't see why you'd even consider Mac hardware for your desktop. In the OP's position, suggesting Apple hardware literally makes no sense. They're not asking for the best hardware that runs MacOS, they're asking for the best hardware for AI. > If I could spend $4k on a non-Apple turn key solution that I could reasonably manage in my house, I would totally consider it. You can't pay Apple $4k for a turnkey solution, either. MacOS is borderline useless for headless inference; Vulkan compute and OpenCL are both MIA, package managers break on regular system updates and don't support rollback, LTS support barely exists, most coreutils are outdated and unmaintained, Asahi features things that MacOS doesn't support and vice-versa... you can't fool me into thinking that's a "turn key solution" any day of the week. If your car requires you to pick a package manager after you turn the engine over, then I really feel sorry for you. The state of MacOS for AI inference is truly no better than what Microsoft did with DirectML. By some accounts it's quite a bit worse.

adam_arthur 2y ago

A Mac Mini with 24GB is ~$800 at the cheapest configuration. I can respect wanting to do a single part upgrade, but if you're using these LLMs for serious work, the price/perf for inferencing is far in favor of using Macs at the moment. You can easily use the MacMini as a hub for running the LLM while you do work on your main computer (and it won't eat up your system resources or turn your primary computer into a heater) I hope that more non-mac PCs come out optimized for high RAM SoC, I'm personally not a huge Apple fan but use them begrudgingly. Also your $900 quote is a used/refurbished GPU. I've had plenty of GPUs burn out on me in the old days, not sure how it is nowadays, but that's a lot to pay for a used part IMO

fragmede 2y ago
if you're doing serious work, performance is more important than getting a good price/perf ratio, and a pair of 3090s is gonna be faster. It depends on your budget, however as that configuration is a bit more expensive, however.
- adam_arthur 2y ago
  Whether performance or cost is more important depends on your use case. Some tasks that an LLM can do very well may not need to be done often, or even particularly quickly (as in my case). e.g. LLM as one step of an ETL-style pipeline Latency of the response really only matters if that response is user facing and is being actively awaited by the user

seanmcdirmid 2y ago

> M4 Max's compute somewhere near the desktop 3060 Ti The only advantage is the M4 Max's ability to have way more VRAM than a 3060 Ti. You won't find many M4 Maxes with just 8 or 16 GB of RAM, and I don't think you can do much except use really small models with a 3060 Ti.

talldayo 2y ago
It's a bit of a moot point when CUDA will run 4 3060Tis in parallel, with further options for paging out to system memory. Since most models (particularly bigger/MOE ones) are sparsely decoded, you can get quite a lot of mileage out of multiple PCIe slots fed with enough bandwidth. There's no doubt in my mind that the PC is the better performer if raw power is your concern. It's far-and-away the better value if you don't need to buy new hardware and only need a GPU. $2,000 of Nvidia GPUs will buy you halfway to an enterprise cluster, $2,000 of Apple hardware will get you a laptop chip with HBM.
- seanmcdirmid 2y ago
  You need a lot of space for that, cooling, and a good fuse that won't trip when you turn it on. I would totally just pay the money for an M4 Ultra MacStudio with 128 GB of RAM (or an M4 Max with 64 GB). It is a much cleaner setup, especially if you aren't interested in image generation (which the Macs are not good at yet). If I could spend $4k on a non-Apple turn key solution that I could reasonably manage in my house, I would totally consider it.
  - talldayo 2y ago
    Well, that's your call. If you're the sort of person that's willing to spend $2,000 on a M4 Ultra (which doesn't quite exist yet but we can pretend it does), then I honest to god do not understand why you'd refuse to spend that same money on a Jetson Orin with the same amount of memory in a smaller footprint with better performance and lower power consumption. Unless you're specifically speccing out a computer for mobile use, the price premium you spend on a Mac isn't for better software or faster hardware. If you can tolerate Linux or Windows, I don't see why you'd even consider Mac hardware for your desktop. In the OP's position, suggesting Apple hardware literally makes no sense. They're not asking for the best hardware that runs MacOS, they're asking for the best hardware for AI. > If I could spend $4k on a non-Apple turn key solution that I could reasonably manage in my house, I would totally consider it. You can't pay Apple $4k for a turnkey solution, either. MacOS is borderline useless for headless inference; Vulkan compute and OpenCL are both MIA, package managers break on regular system updates and don't support rollback, LTS support barely exists, most coreutils are outdated and unmaintained, Asahi features things that MacOS doesn't support and vice-versa... you can't fool me into thinking that's a "turn key solution" any day of the week. If your car requires you to pick a package manager after you turn the engine over, then I really feel sorry for you. The state of MacOS for AI inference is truly no better than what Microsoft did with DirectML. By some accounts it's quite a bit worse.