4 ms·

$9k for three of those? I'm affraid Your math aint mathing. 7-slot motherboards are IMO Threadripper teritory. The CPU alone will set You back at least $1k5, M

$9k for three of those? I'm affraid Your math aint mathing. 7-slot motherboards are IMO Threadripper teritory. The CPU alone will set You back at least $1k5, MB to go along with that is another $1k. Not including the stack of DDR5 that You'll likely throw into that and the PSU / case / coolers to feed it power and evacuate the heat from this monstrosity. That's at least $3000 for the base system "before" You add any GPUs. And at some point I'd like to read someone more enlightened to comment on what the memory bandwidth is likely to be and how exactly will You distribute the PCIe lanes...

frognumber 3y ago
PCIe bandwidth is a limiting factor. For brevity, I did not write an essay. The rest doesn't change the big picture. I spec'ed this out before, and your numbers are a good bit off if you're willing to cut corners. The cheapest way to do the rest is refurbished server, and slightly older parts. That's <<$1000. New parts is a little bit more, but nowhere near $3k. Motherboard can be found for under $500 when a deal comes up. I found the Supermicro MBD-M12SWA-TF-O for that. PCIe risers and cooling can be done on the cheap, if you're willing to DIY. Etc. Power draw is under 3kW. A pair of 1.5kW PSUs will do that, for under $300. It's a mess (you're not getting a formal case), but it works. Very similar systems were built during the bitcoin boom. Look some of those up, and see how they got costs down. That was all about margins. I would like someone more enlightened to comment on PCIe issues as well.
- MezzoDelCammin 3y ago
  True, bitcoin miners have probably figured out quite a few things to which I'm fairly oblivious. I always regarded that movement as a complete waste of computing power. Turns out at least some of their stuff may turn out to be useful for LLMs... As for the specs, I went new Threadripper way simply because that's AFAIK where You'll get the best balance of PCIe 5.0 lanes (for the memory bandwidth between the GPUs) and power. I see how You could spec this out with some used Xeon, but for one I really wouldn't like the headache and another, You're probably getting PCIe 2.0/3.0 tops (if You're trying to cut costs that much). That Supermicro You mention seems to be an interesting compromise. The MB is selling second hand for about ~700EUR in the EU. The Threadripper that goes into it... don't know, can't find them used, but guessing something under 1k EUR as well. Interesting price, but then on top of the risk of using old parts, instead of partitioning PCIe 5.0 (and maybe having the option to go 4.0 and double the count, since Your GPUs are PCIe 4.0 anyway), You're now splitting straight 4.0 lanes to Your GPUs. Half the bandwidth, but it may end up not mattering all that much. Given that most LLMs are memory size / memory bandwidth limited, I probably wouldn't like to risk this. But that's just me...
  - frognumber 3y ago
    A few points: * The Supermicro was on sale for around $500 a few weeks ago. That's an actual price new on Newegg. I had it in my cart (not with intent to purchase now, but spec'ing out for a future purchase). * I've found proper refurbished to be at least as reliable as new, if bought through appropriate channels. What you're looking for is off-lease. Companies lease computers for e.g. 3 years, and then return to Dell. Those computers ran for long enough to be past infant mortality. That very much differs from consumer returns / repairs, which were returned because they were unreliable. I'd never buy a refubished Inspiron, but I very much would buy a refurbished Latitude. * Part of it depends on what you're doing. I'm very much not performance-limited for most of the things I do. If a model takes 2x as long to run, I'll wait twice as long. No big deal. If a model can't run at all, that's where there's a huge difference. A card like the A770, which is around $20/GB, is very exciting to me, if the drivers / documentation / etc. lands. I'll also mention: The scaling here isn't quite as it seems: 1) Two 16GB cards with half the bandwidth per card will have the same theoretical bandwidth as one 32GB card in a faster lane. Ditto for bandwidth to VRAM. Once you start doing the math there, the cheap cards look very good, indeed! 2) If big chunks of data need to flow between cards, you're using much more of that bandwidth. The cheap cards look a lot worse. A lot of that depends on what you're doing, and if I haven't made it clear yet, I don't quite know what I'm doing :)

frognumber 3y ago

PCIe bandwidth is a limiting factor. For brevity, I did not write an essay. The rest doesn't change the big picture. I spec'ed this out before, and your numbers are a good bit off if you're willing to cut corners. The cheapest way to do the rest is refurbished server, and slightly older parts. That's <<$1000. New parts is a little bit more, but nowhere near $3k. Motherboard can be found for under $500 when a deal comes up. I found the Supermicro MBD-M12SWA-TF-O for that. PCIe risers and cooling can be done on the cheap, if you're willing to DIY. Etc. Power draw is under 3kW. A pair of 1.5kW PSUs will do that, for under $300. It's a mess (you're not getting a formal case), but it works. Very similar systems were built during the bitcoin boom. Look some of those up, and see how they got costs down. That was all about margins. I would like someone more enlightened to comment on PCIe issues as well.

MezzoDelCammin 3y ago
True, bitcoin miners have probably figured out quite a few things to which I'm fairly oblivious. I always regarded that movement as a complete waste of computing power. Turns out at least some of their stuff may turn out to be useful for LLMs... As for the specs, I went new Threadripper way simply because that's AFAIK where You'll get the best balance of PCIe 5.0 lanes (for the memory bandwidth between the GPUs) and power. I see how You could spec this out with some used Xeon, but for one I really wouldn't like the headache and another, You're probably getting PCIe 2.0/3.0 tops (if You're trying to cut costs that much). That Supermicro You mention seems to be an interesting compromise. The MB is selling second hand for about ~700EUR in the EU. The Threadripper that goes into it... don't know, can't find them used, but guessing something under 1k EUR as well. Interesting price, but then on top of the risk of using old parts, instead of partitioning PCIe 5.0 (and maybe having the option to go 4.0 and double the count, since Your GPUs are PCIe 4.0 anyway), You're now splitting straight 4.0 lanes to Your GPUs. Half the bandwidth, but it may end up not mattering all that much. Given that most LLMs are memory size / memory bandwidth limited, I probably wouldn't like to risk this. But that's just me...
- frognumber 3y ago
  A few points: * The Supermicro was on sale for around $500 a few weeks ago. That's an actual price new on Newegg. I had it in my cart (not with intent to purchase now, but spec'ing out for a future purchase). * I've found proper refurbished to be at least as reliable as new, if bought through appropriate channels. What you're looking for is off-lease. Companies lease computers for e.g. 3 years, and then return to Dell. Those computers ran for long enough to be past infant mortality. That very much differs from consumer returns / repairs, which were returned because they were unreliable. I'd never buy a refubished Inspiron, but I very much would buy a refurbished Latitude. * Part of it depends on what you're doing. I'm very much not performance-limited for most of the things I do. If a model takes 2x as long to run, I'll wait twice as long. No big deal. If a model can't run at all, that's where there's a huge difference. A card like the A770, which is around $20/GB, is very exciting to me, if the drivers / documentation / etc. lands. I'll also mention: The scaling here isn't quite as it seems: 1) Two 16GB cards with half the bandwidth per card will have the same theoretical bandwidth as one 32GB card in a faster lane. Ditto for bandwidth to VRAM. Once you start doing the math there, the cheap cards look very good, indeed! 2) If big chunks of data need to flow between cards, you're using much more of that bandwidth. The cheap cards look a lot worse. A lot of that depends on what you're doing, and if I haven't made it clear yet, I don't quite know what I'm doing :)