4 ms·

That's more-or-less my experience with AMD, only worse. Critical thing too are burned developers like myself. I'm looking forward to Intel v. NVidia. Arc A770

That's more-or-less my experience with AMD, only worse. Critical thing too are burned developers like myself. I'm looking forward to Intel v. NVidia. Arc A770 is a pretty serious competitor. It's the lowest-cost way to run OPT-175B. Given a 7-slot motherboard, $270 * 7 = $1890 for 112GB of VRAM in one computer. That's sweet. Compute speed would be on-par with top-of-the-line NVidia workstation GPU. Three of those are enough to run the largest open-source LLMs at around $9000. We're just drivers + libraries + documentation away, and Intel is not bad at drivers + libraries + documentation.

vGPU 3y ago
Sure but what’s the energy and overhead tradeoff?
ParetoOptimal 3y ago
> Arc A770 is a pretty serious competitor. It's the lowest-cost way to run OPT-175B. With what software?
sangnoir 3y ago
If AMD can significantly undercut Nvidia on perf/$ like they did to Intel with Zen 1; developers will fall over themselves to improve the software. When the performance price-point is low enough, the pain will be worth it.
MezzoDelCammin 3y ago
$9k for three of those? I'm affraid Your math aint mathing. 7-slot motherboards are IMO Threadripper teritory. The CPU alone will set You back at least $1k5, MB to go along with that is another $1k. Not including the stack of DDR5 that You'll likely throw into that and the PSU / case / coolers to feed it power and evacuate the heat from this monstrosity. That's at least $3000 for the base system "before" You add any GPUs. And at some point I'd like to read someone more enlightened to comment on what the memory bandwidth is likely to be and how exactly will You distribute the PCIe lanes...
- frognumber 3y ago
  PCIe bandwidth is a limiting factor. For brevity, I did not write an essay. The rest doesn't change the big picture. I spec'ed this out before, and your numbers are a good bit off if you're willing to cut corners. The cheapest way to do the rest is refurbished server, and slightly older parts. That's <<$1000. New parts is a little bit more, but nowhere near $3k. Motherboard can be found for under $500 when a deal comes up. I found the Supermicro MBD-M12SWA-TF-O for that. PCIe risers and cooling can be done on the cheap, if you're willing to DIY. Etc. Power draw is under 3kW. A pair of 1.5kW PSUs will do that, for under $300. It's a mess (you're not getting a formal case), but it works. Very similar systems were built during the bitcoin boom. Look some of those up, and see how they got costs down. That was all about margins. I would like someone more enlightened to comment on PCIe issues as well.
  - MezzoDelCammin 3y ago
    True, bitcoin miners have probably figured out quite a few things to which I'm fairly oblivious. I always regarded that movement as a complete waste of computing power. Turns out at least some of their stuff may turn out to be useful for LLMs... As for the specs, I went new Threadripper way simply because that's AFAIK where You'll get the best balance of PCIe 5.0 lanes (for the memory bandwidth between the GPUs) and power. I see how You could spec this out with some used Xeon, but for one I really wouldn't like the headache and another, You're probably getting PCIe 2.0/3.0 tops (if You're trying to cut costs that much). That Supermicro You mention seems to be an interesting compromise. The MB is selling second hand for about ~700EUR in the EU. The Threadripper that goes into it... don't know, can't find them used, but guessing something under 1k EUR as well. Interesting price, but then on top of the risk of using old parts, instead of partitioning PCIe 5.0 (and maybe having the option to go 4.0 and double the count, since Your GPUs are PCIe 4.0 anyway), You're now splitting straight 4.0 lanes to Your GPUs. Half the bandwidth, but it may end up not mattering all that much. Given that most LLMs are memory size / memory bandwidth limited, I probably wouldn't like to risk this. But that's just me...
    - frognumber 3y ago
JonChesterfield 3y ago
ATX power supplies cap out at 2kW or so and single slot GPUs require water cooling which moves your price estimate significantly. Also difficult to plumb in adjacent cards. I think the realistic maximum is four ~300W cards per system, and that still needs water. Your point stands - GPU compute on commodity hardware is very cheap. $10k per 4U quad GPU box is probably the ballpark.
evilos 3y ago
Intel as a whole seems to be in pretty big trouble and rumor is that the GPU org is a dumpster fire in particular. I'd bet it's more likely than not that intel's GPU experiment ends within a year or two, unfortunately. Which is a crying shame because the space could really use more competition, instead of less.

vGPU 3y ago

Sure but what’s the energy and overhead tradeoff?

ParetoOptimal 3y ago

> Arc A770 is a pretty serious competitor. It's the lowest-cost way to run OPT-175B. With what software?

sangnoir 3y ago

If AMD can significantly undercut Nvidia on perf/$ like they did to Intel with Zen 1; developers will fall over themselves to improve the software. When the performance price-point is low enough, the pain will be worth it.

MezzoDelCammin 3y ago

$9k for three of those? I'm affraid Your math aint mathing. 7-slot motherboards are IMO Threadripper teritory. The CPU alone will set You back at least $1k5, MB to go along with that is another $1k. Not including the stack of DDR5 that You'll likely throw into that and the PSU / case / coolers to feed it power and evacuate the heat from this monstrosity. That's at least $3000 for the base system "before" You add any GPUs. And at some point I'd like to read someone more enlightened to comment on what the memory bandwidth is likely to be and how exactly will You distribute the PCIe lanes...

frognumber 3y ago
PCIe bandwidth is a limiting factor. For brevity, I did not write an essay. The rest doesn't change the big picture. I spec'ed this out before, and your numbers are a good bit off if you're willing to cut corners. The cheapest way to do the rest is refurbished server, and slightly older parts. That's <<$1000. New parts is a little bit more, but nowhere near $3k. Motherboard can be found for under $500 when a deal comes up. I found the Supermicro MBD-M12SWA-TF-O for that. PCIe risers and cooling can be done on the cheap, if you're willing to DIY. Etc. Power draw is under 3kW. A pair of 1.5kW PSUs will do that, for under $300. It's a mess (you're not getting a formal case), but it works. Very similar systems were built during the bitcoin boom. Look some of those up, and see how they got costs down. That was all about margins. I would like someone more enlightened to comment on PCIe issues as well.
- MezzoDelCammin 3y ago
  True, bitcoin miners have probably figured out quite a few things to which I'm fairly oblivious. I always regarded that movement as a complete waste of computing power. Turns out at least some of their stuff may turn out to be useful for LLMs... As for the specs, I went new Threadripper way simply because that's AFAIK where You'll get the best balance of PCIe 5.0 lanes (for the memory bandwidth between the GPUs) and power. I see how You could spec this out with some used Xeon, but for one I really wouldn't like the headache and another, You're probably getting PCIe 2.0/3.0 tops (if You're trying to cut costs that much). That Supermicro You mention seems to be an interesting compromise. The MB is selling second hand for about ~700EUR in the EU. The Threadripper that goes into it... don't know, can't find them used, but guessing something under 1k EUR as well. Interesting price, but then on top of the risk of using old parts, instead of partitioning PCIe 5.0 (and maybe having the option to go 4.0 and double the count, since Your GPUs are PCIe 4.0 anyway), You're now splitting straight 4.0 lanes to Your GPUs. Half the bandwidth, but it may end up not mattering all that much. Given that most LLMs are memory size / memory bandwidth limited, I probably wouldn't like to risk this. But that's just me...
  - frognumber 3y ago

JonChesterfield 3y ago

ATX power supplies cap out at 2kW or so and single slot GPUs require water cooling which moves your price estimate significantly. Also difficult to plumb in adjacent cards. I think the realistic maximum is four ~300W cards per system, and that still needs water. Your point stands - GPU compute on commodity hardware is very cheap. $10k per 4U quad GPU box is probably the ballpark.

evilos 3y ago

Intel as a whole seems to be in pretty big trouble and rumor is that the GPU org is a dumpster fire in particular. I'd bet it's more likely than not that intel's GPU experiment ends within a year or two, unfortunately. Which is a crying shame because the space could really use more competition, instead of less.