3 ms·

True, bitcoin miners have probably figured out quite a few things to which I'm fairly oblivious. I always regarded that movement as a complete waste of computin

True, bitcoin miners have probably figured out quite a few things to which I'm fairly oblivious. I always regarded that movement as a complete waste of computing power. Turns out at least some of their stuff may turn out to be useful for LLMs... As for the specs, I went new Threadripper way simply because that's AFAIK where You'll get the best balance of PCIe 5.0 lanes (for the memory bandwidth between the GPUs) and power. I see how You could spec this out with some used Xeon, but for one I really wouldn't like the headache and another, You're probably getting PCIe 2.0/3.0 tops (if You're trying to cut costs that much). That Supermicro You mention seems to be an interesting compromise. The MB is selling second hand for about ~700EUR in the EU. The Threadripper that goes into it... don't know, can't find them used, but guessing something under 1k EUR as well. Interesting price, but then on top of the risk of using old parts, instead of partitioning PCIe 5.0 (and maybe having the option to go 4.0 and double the count, since Your GPUs are PCIe 4.0 anyway), You're now splitting straight 4.0 lanes to Your GPUs. Half the bandwidth, but it may end up not mattering all that much. Given that most LLMs are memory size / memory bandwidth limited, I probably wouldn't like to risk this. But that's just me...

frognumber 3y ago
A few points: * The Supermicro was on sale for around $500 a few weeks ago. That's an actual price new on Newegg. I had it in my cart (not with intent to purchase now, but spec'ing out for a future purchase). * I've found proper refurbished to be at least as reliable as new, if bought through appropriate channels. What you're looking for is off-lease. Companies lease computers for e.g. 3 years, and then return to Dell. Those computers ran for long enough to be past infant mortality. That very much differs from consumer returns / repairs, which were returned because they were unreliable. I'd never buy a refubished Inspiron, but I very much would buy a refurbished Latitude. * Part of it depends on what you're doing. I'm very much not performance-limited for most of the things I do. If a model takes 2x as long to run, I'll wait twice as long. No big deal. If a model can't run at all, that's where there's a huge difference. A card like the A770, which is around $20/GB, is very exciting to me, if the drivers / documentation / etc. lands. I'll also mention: The scaling here isn't quite as it seems: 1) Two 16GB cards with half the bandwidth per card will have the same theoretical bandwidth as one 32GB card in a faster lane. Ditto for bandwidth to VRAM. Once you start doing the math there, the cheap cards look very good, indeed! 2) If big chunks of data need to flow between cards, you're using much more of that bandwidth. The cheap cards look a lot worse. A lot of that depends on what you're doing, and if I haven't made it clear yet, I don't quite know what I'm doing :)

frognumber 3y ago

A few points: * The Supermicro was on sale for around $500 a few weeks ago. That's an actual price new on Newegg. I had it in my cart (not with intent to purchase now, but spec'ing out for a future purchase). * I've found proper refurbished to be at least as reliable as new, if bought through appropriate channels. What you're looking for is off-lease. Companies lease computers for e.g. 3 years, and then return to Dell. Those computers ran for long enough to be past infant mortality. That very much differs from consumer returns / repairs, which were returned because they were unreliable. I'd never buy a refubished Inspiron, but I very much would buy a refurbished Latitude. * Part of it depends on what you're doing. I'm very much not performance-limited for most of the things I do. If a model takes 2x as long to run, I'll wait twice as long. No big deal. If a model can't run at all, that's where there's a huge difference. A card like the A770, which is around $20/GB, is very exciting to me, if the drivers / documentation / etc. lands. I'll also mention: The scaling here isn't quite as it seems: 1) Two 16GB cards with half the bandwidth per card will have the same theoretical bandwidth as one 32GB card in a faster lane. Ditto for bandwidth to VRAM. Once you start doing the math there, the cheap cards look very good, indeed! 2) If big chunks of data need to flow between cards, you're using much more of that bandwidth. The cheap cards look a lot worse. A lot of that depends on what you're doing, and if I haven't made it clear yet, I don't quite know what I'm doing :)