3 ms·

Yes, you can. Ideally though, you want to minimize the number of cards and maximize the amount of memory in each card. Using multiple cards is one of the thing

Yes, you can. Ideally though, you want to minimize the number of cards and maximize the amount of memory in each card. Using multiple cards is one of the things that the models and software that Unsloth releases does really well in terms of ease of use and relatively good performance.

jambalaya8 2mo ago
How many do you recommend?
- strictnein 2mo ago
  I've run 4, 6, and 8. Adding more video cards increases total available memory, so you can load larger models, but there are definitely some drawbacks. More cards = more communication over PCIe. The prompts don't come in and just get magically split between each card, they move sequentially through them. Also, with 2x24GB cards you don't really have 48GB of usable memory to load a model, closer to ~42GB + context. And then there are power concerns, motherboard limitations (PCIe slots and lanes - a lot of motherboards with multiple 16x PCIe slots don't actually have 16x lanes to each of those slots), and more. 8x GPUs are going to easily draw 2000W on their own, if not substantially more. You'll need wiring and a circuit that can support 3000W without a risk of starting a fire in your wall. For $5k, a single 32GB 5090 might be a better choice for a lot of people versus 4x3090s with 24GB each. It will definitely perform substantially better on smaller 27B models. For hardware: A good motherboard with lots of PCIe lanes (7x full 16x PCIe 4.0), DDR4 support, etc: https://www.asus.com/us/motherboards-components/motherboards/workstation/pro-ws-wrx80e-sage-se-wifi/techspec/ https://www.asus.com/us/motherboards-components/motherboards... Add in a 3xxx series Threadripper PRO, 128 or 256GB of DDR4 (going higher becomes really expensive), and a ~1400 watt power supply. You can underpower/undervolt Nvidia cards really easily, and capping them at 250W loses you minimal performance.

jambalaya8 2mo ago

How many do you recommend?

strictnein 2mo ago
I've run 4, 6, and 8. Adding more video cards increases total available memory, so you can load larger models, but there are definitely some drawbacks. More cards = more communication over PCIe. The prompts don't come in and just get magically split between each card, they move sequentially through them. Also, with 2x24GB cards you don't really have 48GB of usable memory to load a model, closer to ~42GB + context. And then there are power concerns, motherboard limitations (PCIe slots and lanes - a lot of motherboards with multiple 16x PCIe slots don't actually have 16x lanes to each of those slots), and more. 8x GPUs are going to easily draw 2000W on their own, if not substantially more. You'll need wiring and a circuit that can support 3000W without a risk of starting a fire in your wall. For $5k, a single 32GB 5090 might be a better choice for a lot of people versus 4x3090s with 24GB each. It will definitely perform substantially better on smaller 27B models. For hardware: A good motherboard with lots of PCIe lanes (7x full 16x PCIe 4.0), DDR4 support, etc: https://www.asus.com/us/motherboards-components/motherboards/workstation/pro-ws-wrx80e-sage-se-wifi/techspec/ https://www.asus.com/us/motherboards-components/motherboards... Add in a 3xxx series Threadripper PRO, 128 or 256GB of DDR4 (going higher becomes really expensive), and a ~1400 watt power supply. You can underpower/undervolt Nvidia cards really easily, and capping them at 250W loses you minimal performance.