3 ms·

This might sound crazy but would it make more sense for me to buy a Mac Studio instead of the new GTX40 series if I just want to dabble in some fun ML like stab

This might sound crazy but would it make more sense for me to buy a Mac Studio instead of the new GTX40 series if I just want to dabble in some fun ML like stable diffusion? I don’t plan on training anything crazy on my own, even if I do, it’ll be fine tuning rather than scratch.

aseipp 4y ago
No, go pick up a 3090 Ti from Best Buy instead. It's $999 right now -- over half off, maybe more with this announcement -- and has 24GB of VRAM, and can chew through Stable Diffusion requests in seconds versus dozens of seconds or minutes for alternatives. And because it's an nvidia card, you'll be in a much better position to troubleshoot problems or whatever if you want to retune/retrain models or get your feet wet. Or, you know. Shove it into another PC and play a video game. That said the Mac Studio is a pretty badass machine. Don't let me dissuade you from one if you just want it. But it's a worse choice if you just want to dabble in ML stuff, IMO. Staying with the well trodden road will help you more when you're new to it all.
- syntaxing 4y ago
  Does regular RAM matter if I want to run stable diffusion? I only have 16GB RAM (but obviously upgradable). If I load the weights onto the VRAM, does it work SSD > RAM > VRAM or does it bypass the RAM?
  - aseipp 4y ago
    Yes, Stable Diffusion performance will improve the more VRAM you give it. This is generally true of almost every ML model, because ML models are often limited by bandwidth. There is a constant need to shove more data into the accelerator, but PCIe is very slow compared to data in VRAM. If you constantly had to push data over PCIe, your GPU would run at 5% of its max performance, and spend 95% of its time waiting on PCIe transfers. Instead, you load a bunch of data over PCIE at once into VRAM, as much as possible, and then you begin working. So the more VRAM you have, the more data you have "in hand" -- so you can have a bigger batch size for your training iterations, for example. So one of the things various releases of Stable Diffusion have tried to optimize, for example, is VRAM usage, because that's the #1 limiting factor for most people. Modern GPUs use virtual memory so they can see very large amounts of data and the GPU and OS will work together to page data in and out, like normal RAM. But if you want good performance, you need to work out of VRAM, so you need as much of it as possible, so you aren't paging data in and out. (More specifically to answer your question, yes it will page data from SSD into RAM and then into VRAM; this is because the data must pass through the memory controller in practice on most designs to go over PCIe, but if you are using fancier Tesla/Quadro cards, they can actually do P2P DMA transfers and go directly from SSD -> GPU, but this requires special care and support from all pieces of hardware inbetween.) Note that in a funny twist of fate, the M1 series from Apple kind of has an advantage here. They use very high-bandwidth LPDDR5X for their designs, and the GPU and CPU share that. Which means that in theory, the Mac Studio for example could have astronomically large amounts of VRAM dedicated to a task like ML. So why not use a Mac? Because the hardware is only good as the software you run it on, and training/tuning ML software on Macs is simply nowhere near as well polished as Linux/Nvidia. That's all it comes down to. And I say "well polished" very liberally; in practice it's all a bit of a hodge-podge nightmare because the job is very hodge-podge...
  - lostmsu 4y ago
    You don't need to care about this.
  - moandcompany 4y ago
    The model checkpoint file (i.e. weights) being used is read from disk, rather than from system memory afaik. 16GB of system RAM is more than sufficient.
  - sdflhasjd 4y ago
    It is fairly RAM heavy, but 16 should be plenty
dougmwne 4y ago
For ML work you can rent a cloud GPU easily enough. Most people use Google Colab and get a bucket of free credits that last then awhile.

aseipp 4y ago

No, go pick up a 3090 Ti from Best Buy instead. It's $999 right now -- over half off, maybe more with this announcement -- and has 24GB of VRAM, and can chew through Stable Diffusion requests in seconds versus dozens of seconds or minutes for alternatives. And because it's an nvidia card, you'll be in a much better position to troubleshoot problems or whatever if you want to retune/retrain models or get your feet wet. Or, you know. Shove it into another PC and play a video game. That said the Mac Studio is a pretty badass machine. Don't let me dissuade you from one if you just want it. But it's a worse choice if you just want to dabble in ML stuff, IMO. Staying with the well trodden road will help you more when you're new to it all.

syntaxing 4y ago
Does regular RAM matter if I want to run stable diffusion? I only have 16GB RAM (but obviously upgradable). If I load the weights onto the VRAM, does it work SSD > RAM > VRAM or does it bypass the RAM?
- aseipp 4y ago
  Yes, Stable Diffusion performance will improve the more VRAM you give it. This is generally true of almost every ML model, because ML models are often limited by bandwidth. There is a constant need to shove more data into the accelerator, but PCIe is very slow compared to data in VRAM. If you constantly had to push data over PCIe, your GPU would run at 5% of its max performance, and spend 95% of its time waiting on PCIe transfers. Instead, you load a bunch of data over PCIE at once into VRAM, as much as possible, and then you begin working. So the more VRAM you have, the more data you have "in hand" -- so you can have a bigger batch size for your training iterations, for example. So one of the things various releases of Stable Diffusion have tried to optimize, for example, is VRAM usage, because that's the #1 limiting factor for most people. Modern GPUs use virtual memory so they can see very large amounts of data and the GPU and OS will work together to page data in and out, like normal RAM. But if you want good performance, you need to work out of VRAM, so you need as much of it as possible, so you aren't paging data in and out. (More specifically to answer your question, yes it will page data from SSD into RAM and then into VRAM; this is because the data must pass through the memory controller in practice on most designs to go over PCIe, but if you are using fancier Tesla/Quadro cards, they can actually do P2P DMA transfers and go directly from SSD -> GPU, but this requires special care and support from all pieces of hardware inbetween.) Note that in a funny twist of fate, the M1 series from Apple kind of has an advantage here. They use very high-bandwidth LPDDR5X for their designs, and the GPU and CPU share that. Which means that in theory, the Mac Studio for example could have astronomically large amounts of VRAM dedicated to a task like ML. So why not use a Mac? Because the hardware is only good as the software you run it on, and training/tuning ML software on Macs is simply nowhere near as well polished as Linux/Nvidia. That's all it comes down to. And I say "well polished" very liberally; in practice it's all a bit of a hodge-podge nightmare because the job is very hodge-podge...
- lostmsu 4y ago
  You don't need to care about this.
- moandcompany 4y ago
  The model checkpoint file (i.e. weights) being used is read from disk, rather than from system memory afaik. 16GB of system RAM is more than sufficient.
- sdflhasjd 4y ago
  It is fairly RAM heavy, but 16 should be plenty

dougmwne 4y ago

For ML work you can rent a cloud GPU easily enough. Most people use Google Colab and get a bucket of free credits that last then awhile.