4 ms·

Side question: is there a benefit to doing this on an MacBook vs PC ?

lagniappe 3y ago
Personally speaking I never realized how much fan noise and hot air coming from the machine distracted me until it wasn't there anymore.
brucethemoose2 3y ago
Macs have a larger iGPU and faster RAM than pretty much any comparable PC, which are precisely the two things you want for LLMs. Also, lots of people only own a Mac. You'd have to be kinda crazy to buy a Mac explicitly for running/training genai though.
- lostmsu 3y ago
  I don't think you want iGPU for LLMs. 4090 + RAM offload would be many times faster.
  - brucethemoose2 3y ago
    3090 + RAM offload is not very fast on my desktop. Its fine, but the speed hit is large. And I was thinking regular laptops as a baseline vs the base M1/M2.
    - lostmsu 3y ago
      MacBooks don't have enough RAM to train without offloading to SSD which would be even slower.
duskwuff 3y ago
Yes, and it revolves around memory. If you're running code on the CPU, Apple Silicon systems have significantly higher memory bandwidth than most x86 systems -- up to 800 GB/sec on M2 Ultra. If you're running code on the GPU, an Apple Silicon system can be configured with up to 192 GB of unified memory, whereas most discrete graphics cards top out around 16-24 GB of VRAM. (There are a few larger compute-focused cards like Nvidia A100, but they're incredibly expensive.)
- omneity 3y ago
  There’s no _single_ consumer or datacenter GPU that is currently breaking 100GB VRAM. In this regard, the Apple offering is pretty unique, but their compute capability is lagging. Also the 800GB/s figure for memory bandwidth is impressive ... for a CPU. GPUs regularly hit 2TB/s or more. TL;DR on Apple for ML: Bigger models, slower calculations.
  - 65a 3y ago
    Not quite, there are 128GiB AMD instinct parts, though the prices are very...enterprise: https://www.amd.com/en/products/server-accelerators/instinct-mi250 https://www.amd.com/en/products/server-accelerators/instinct...
    - omneity 3y ago
      Wow that's cool. I didn't know about it, thanks for sharing! Side note, it looks like the AMD MI250 supports FP16, and the raw TFLOPS is ~15% higher than an Nvidia A100 80GB, with a significant advantage for AMD in terms of memory bandwidth. The price range for the two cards seem to roughly overlap. It seems this card is a pretty good deal barring crazy software edge cases. @AMD, if you ever see this, I would be interested to test your card and publish a complete writeup. I'm currently fine-tuning LLMs on A100/H100 and have a good reference point.

Personally speaking I never realized how much fan noise and hot air coming from the machine distracted me until it wasn't there anymore.

brucethemoose2 3y ago

Macs have a larger iGPU and faster RAM than pretty much any comparable PC, which are precisely the two things you want for LLMs. Also, lots of people only own a Mac. You'd have to be kinda crazy to buy a Mac explicitly for running/training genai though.

lostmsu 3y ago
I don't think you want iGPU for LLMs. 4090 + RAM offload would be many times faster.
- brucethemoose2 3y ago
  3090 + RAM offload is not very fast on my desktop. Its fine, but the speed hit is large. And I was thinking regular laptops as a baseline vs the base M1/M2.
  - lostmsu 3y ago
    MacBooks don't have enough RAM to train without offloading to SSD which would be even slower.

duskwuff 3y ago

Yes, and it revolves around memory. If you're running code on the CPU, Apple Silicon systems have significantly higher memory bandwidth than most x86 systems -- up to 800 GB/sec on M2 Ultra. If you're running code on the GPU, an Apple Silicon system can be configured with up to 192 GB of unified memory, whereas most discrete graphics cards top out around 16-24 GB of VRAM. (There are a few larger compute-focused cards like Nvidia A100, but they're incredibly expensive.)

omneity 3y ago
There’s no _single_ consumer or datacenter GPU that is currently breaking 100GB VRAM. In this regard, the Apple offering is pretty unique, but their compute capability is lagging. Also the 800GB/s figure for memory bandwidth is impressive ... for a CPU. GPUs regularly hit 2TB/s or more. TL;DR on Apple for ML: Bigger models, slower calculations.
- 65a 3y ago
  Not quite, there are 128GiB AMD instinct parts, though the prices are very...enterprise: https://www.amd.com/en/products/server-accelerators/instinct-mi250 https://www.amd.com/en/products/server-accelerators/instinct...
  - omneity 3y ago
    Wow that's cool. I didn't know about it, thanks for sharing! Side note, it looks like the AMD MI250 supports FP16, and the raw TFLOPS is ~15% higher than an Nvidia A100 80GB, with a significant advantage for AMD in terms of memory bandwidth. The price range for the two cards seem to roughly overlap. It seems this card is a pretty good deal barring crazy software edge cases. @AMD, if you ever see this, I would be interested to test your card and publish a complete writeup. I'm currently fine-tuning LLMs on A100/H100 and have a good reference point.