4 ms·

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super

by Tepix 1y ago

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard leads to the "ASUS ROG Strix X670E-E Gaming" model. This model can use the 2nd PCIe 5.0 port at only x4 speeds. The RTX 3090 can only do PCIe 4.0 of course so it will run at PCIe 4.0 x4. If you choose a desktop mainboard for having two GPUs, make sure it can run at PCIe x8 speeds when using both GPU slots! Having NVLink between the GPUs is not a replacement for having a fast connection between the CPU+RAM and the GPU and its VRAM. 4) Despite having a last-modified date of September 22nd, he is using his rig mostly with rather outdated or small LLMs and his benchmarks do not mention their quantization, which makes them useless. Also they seem not to be benchmarks at all, but "estimates". Perhaps the headline should be changed to reflect this?

rtaylorgarlock 1y ago
[flagged]
- DecentShoes 1y ago
  Horrible comment and attitude. People are trying to quote you for legitimate comment and criticism. This alone was enough for me to close the tab with your blog and ignore anything else you're going to say.
  - gambiting 1y ago
    That's not the author(I don't think?) just a random troll
- prmoustache 1y ago
  I have js enabled and I can copy text on this page.
  - Tepix 1y ago
    In general I can too, but try copying items from the "key specifications". Or perhaps I just had the impression because you can't mark text because I can't tell which text is marked and which isn't when marking text under "Key Specifications". Mea culpa.
    - prmoustache 1y ago
      yeah the selection is dark grey over black so it is not super visible but you can copy text.
jychang 1y ago
Yeah, this page seems to be not great for beginners and also useless for people with experience. A 2x 3090 build is okay for inference, but even with nvlink you're a bit handicapped for training. You're much better off with getting a 4090 48GB from China for $2.5k and just using that. Example: https://www.alibaba.com/trade/search?keywords=4090+48gb&pricef=2100&pricet=3000 https://www.alibaba.com/trade/search?keywords=4090+48gb&pric... Also, this phrasing is concerning: > WARNING - these components don't fit if you try to copy this build. The bottom GPU is resting on the Arctic p12 slim fans at the bottom of the case and pushing up on the GPU. Also the top arctic p14 Max fans don't have mounting points for half of their screw holes, and are in place by being very tightly wedged against the motherboard, case, and PSU. Also, there's probably way too much pressure on the pcie cables coming off the gpus when you close the glass.
- Gracana 1y ago
  $2.5k is about $1k more than you'd spend on a pair of 3090s, and people I know who've bought blower 4090s say they sound like hair driers.
  - dotnet00 1y ago
    Blowers are loud, but they're easier to pack together, particularly given how most motherboards don't seem to space their two slots sufficiently to accomodate the massive coolers on recent GPUs.
    - mertleee 1y ago
      I can't wait for blower 3090s from China / MSI to get cheap (although I fear this may never happen)
- nxobject 1y ago
  Are the Alibaba 4090s modded to reach 48GB VRAM? (I ask only to figure how why they're that cheap...)
  - 1y ago
glax 1y ago
Sorry for going off topic. But your insight will be helpful on my build I'm thinking about a low budget system, which will be using 1.X99 D8 MAX LGA2011-3 Motherboard - It has 4 pcie 3.0 x16 slots, dual cpu socket. They are priced around $260 with both the cpu 2. 4X AMD MI50 32G cards - They are old now, but they have 32 gigs of vram and also can be sources at $110 each The whole setup would not cost more than $1000, is it a right build ? or something more performant can be built within this budget ?
- juliangoldsmith 1y ago
  I'd use caution with the Mi50s. I bought a 16GB one on eBay a while back and it's been completely unusable. It seems to be a Radeon VII on an Mi50 board, which should technically work. It immediately hangs the first time an OpenCL kernel is run, and doesn't come back up until I reboot. It's possible my issues are due to Mesa or driver config, but I'd strongly recommend buying one to test before going all in. There are a lot of cheap SXM2 V100s and adapter boards out now, which should perform very well. The adapters unfortunately weren't available when I bought my hardware, or I would have scooped up several.
  - BizarroLand 1y ago
    I've seen the sxm2 (x2) with pci extension cards out on ebay for like $350. The 32gb v100s with heatsink are like $600 each, so that would be $1500 or so for a one-off 64gb gpu that is less overall performant than a single 3090.
- OakNinja 1y ago
  Better to buy one used 3090 than those old cards. Everything is not vram. Or, you can do nothing without vram but you can’t do anything with just vram. To use the second pair of pcie slots, you _must_ have two cpus installed. Just saying in case someone finds a board with just one cpu socket populated.
danparsonson 1y ago
> None of them can do PCIe x8 when using two GPUs. Is that important for this workload? I thought most of the effort was spent processing data on the card rather than moving data on or off of it?
fkyoureadthedoc 1y ago
> The page is blocking cut & paste. Super annoying! I've been running Don't F* With Paste* for years for this https://chromewebstore.google.com/detail/dont-f-with-paste/nkgllhigpcljnhoakjkgaieabnkmgdkb https://chromewebstore.google.com/detail/dont-f-with-paste/n...
- jgalt212 1y ago
  Interesting. I guess our content-based marketing pages need to move to canvas-based rendering. That's probably bum too. Straight to serving up jpgs.
  - vidarh 1y ago
    > Straight to serving up jpgs. Back in my Amiga-days we had PowerSnap[1] which did the bargain basement version of OCR: Check the font settings of the window you wanted to cut and paste from, and try to match the font to the bitmap, to let you copy and paste from apps that didn't support it, or from UI element you normally couldn't. These days, just throwing the image at an AI model would be far more resilient... I think we've gotten to the point where it would be hard to compose an image that humans can read but an AI model can't, and easy to compose an image an AI can read but humans can't, so I suspect the only option for your marketing department will be to try to prompt inject the AI into buying your product. (Oh, look, I have written nearly this same comment once before, 11 years ago, on HN[2] - I was wrong about how it worked, and Orgre was right, and my follow up reply appears to be closer to what it actually does) [1] https://aminet.net/package/util/cdity/PowerSnap22a https://aminet.net/package/util/cdity/PowerSnap22a [2] https://news.ycombinator.com/item?id=7631161 https://news.ycombinator.com/item?id=7631161
  - 93po 1y ago
    thankfully most web browsing will be done by LLMs soon and that won't stop them, good riddance to the mess of a web that google has created
    - jgalt212 1y ago
      dead Internet for realz
hengheng 1y ago
I've also learned the hard way to Google "AM4 main board tier list" before buying. Some boards can run a 5950X in name only, while others can comfortably run it close to double its spec power all day. VRMs are a real differentiator for this tier of hardware. (If anyone can comment on the airflow required for 400-500W Epyc CPUs with the tiny VRM heatsinks that Supermicro uses, I'm all ears.)
ericdotlee 1y ago
Any reason you wouldn't opt for the 4090 or 5090?
- Instantix 1y ago
  3090 second hand can be found at something like $600.
yjftsjthsd-h 1y ago
> 3) The shopping link for the mainboard leads to the "ASUS ROG Strix X670E-E Gaming" model. This model can use the 2nd PCIe 5.0 port at only x4 speeds. The RTX 3090 can only do PCIe 4.0 of course so it will run at PCIe 4.0 x4. If you choose a desktop mainboard for having two GPUs, make sure it can run at PCIe x8 speeds when using both GPU slots! Having NVLink between the GPUs is not a replacement for having a fast connection between the CPU+RAM and the GPU and its VRAM. Forgive a noob question: I thought the connection to the GPU was actually fairly unimportant once the model was loaded, because sending input to the model and getting a response is low bandwidth? So it might matter if you're changing models a lot or doing a model that can work on video, but otherwise I thought it didn't really matter.
- Tepix 1y ago
  In general, if all you do is inference with a model that’s in VRAM, you’re right. OTOH it’s simply a matter of picking the right mainboard. If you have one of those sweet new MoE models that won‘t completely fit in your VRAM, offloading means you want PCIe bandwidth, because it will be a bottleneck. Also swapping between LLMs will be faster.

by Tepix 1y ago

rtaylorgarlock 1y ago

[flagged]

DecentShoes 1y ago
Horrible comment and attitude. People are trying to quote you for legitimate comment and criticism. This alone was enough for me to close the tab with your blog and ignore anything else you're going to say.
- gambiting 1y ago
  That's not the author(I don't think?) just a random troll
prmoustache 1y ago
I have js enabled and I can copy text on this page.
- Tepix 1y ago
  In general I can too, but try copying items from the "key specifications". Or perhaps I just had the impression because you can't mark text because I can't tell which text is marked and which isn't when marking text under "Key Specifications". Mea culpa.
  - prmoustache 1y ago
    yeah the selection is dark grey over black so it is not super visible but you can copy text.

jychang 1y ago

Yeah, this page seems to be not great for beginners and also useless for people with experience. A 2x 3090 build is okay for inference, but even with nvlink you're a bit handicapped for training. You're much better off with getting a 4090 48GB from China for $2.5k and just using that. Example: https://www.alibaba.com/trade/search?keywords=4090+48gb&pricef=2100&pricet=3000 https://www.alibaba.com/trade/search?keywords=4090+48gb&pric... Also, this phrasing is concerning: > WARNING - these components don't fit if you try to copy this build. The bottom GPU is resting on the Arctic p12 slim fans at the bottom of the case and pushing up on the GPU. Also the top arctic p14 Max fans don't have mounting points for half of their screw holes, and are in place by being very tightly wedged against the motherboard, case, and PSU. Also, there's probably way too much pressure on the pcie cables coming off the gpus when you close the glass.

Gracana 1y ago
$2.5k is about $1k more than you'd spend on a pair of 3090s, and people I know who've bought blower 4090s say they sound like hair driers.
- dotnet00 1y ago
  Blowers are loud, but they're easier to pack together, particularly given how most motherboards don't seem to space their two slots sufficiently to accomodate the massive coolers on recent GPUs.
  - mertleee 1y ago
    I can't wait for blower 3090s from China / MSI to get cheap (although I fear this may never happen)
nxobject 1y ago
Are the Alibaba 4090s modded to reach 48GB VRAM? (I ask only to figure how why they're that cheap...)
- 1y ago

glax 1y ago

Sorry for going off topic. But your insight will be helpful on my build I'm thinking about a low budget system, which will be using 1.X99 D8 MAX LGA2011-3 Motherboard - It has 4 pcie 3.0 x16 slots, dual cpu socket. They are priced around $260 with both the cpu 2. 4X AMD MI50 32G cards - They are old now, but they have 32 gigs of vram and also can be sources at $110 each The whole setup would not cost more than $1000, is it a right build ? or something more performant can be built within this budget ?

juliangoldsmith 1y ago
I'd use caution with the Mi50s. I bought a 16GB one on eBay a while back and it's been completely unusable. It seems to be a Radeon VII on an Mi50 board, which should technically work. It immediately hangs the first time an OpenCL kernel is run, and doesn't come back up until I reboot. It's possible my issues are due to Mesa or driver config, but I'd strongly recommend buying one to test before going all in. There are a lot of cheap SXM2 V100s and adapter boards out now, which should perform very well. The adapters unfortunately weren't available when I bought my hardware, or I would have scooped up several.
- BizarroLand 1y ago
  I've seen the sxm2 (x2) with pci extension cards out on ebay for like $350. The 32gb v100s with heatsink are like $600 each, so that would be $1500 or so for a one-off 64gb gpu that is less overall performant than a single 3090.
OakNinja 1y ago
Better to buy one used 3090 than those old cards. Everything is not vram. Or, you can do nothing without vram but you can’t do anything with just vram. To use the second pair of pcie slots, you _must_ have two cpus installed. Just saying in case someone finds a board with just one cpu socket populated.

danparsonson 1y ago

> None of them can do PCIe x8 when using two GPUs. Is that important for this workload? I thought most of the effort was spent processing data on the card rather than moving data on or off of it?

fkyoureadthedoc 1y ago

> The page is blocking cut & paste. Super annoying! I've been running Don't F* With Paste* for years for this https://chromewebstore.google.com/detail/dont-f-with-paste/nkgllhigpcljnhoakjkgaieabnkmgdkb https://chromewebstore.google.com/detail/dont-f-with-paste/n...

jgalt212 1y ago
Interesting. I guess our content-based marketing pages need to move to canvas-based rendering. That's probably bum too. Straight to serving up jpgs.
- vidarh 1y ago
  > Straight to serving up jpgs. Back in my Amiga-days we had PowerSnap[1] which did the bargain basement version of OCR: Check the font settings of the window you wanted to cut and paste from, and try to match the font to the bitmap, to let you copy and paste from apps that didn't support it, or from UI element you normally couldn't. These days, just throwing the image at an AI model would be far more resilient... I think we've gotten to the point where it would be hard to compose an image that humans can read but an AI model can't, and easy to compose an image an AI can read but humans can't, so I suspect the only option for your marketing department will be to try to prompt inject the AI into buying your product. (Oh, look, I have written nearly this same comment once before, 11 years ago, on HN[2] - I was wrong about how it worked, and Orgre was right, and my follow up reply appears to be closer to what it actually does) [1] https://aminet.net/package/util/cdity/PowerSnap22a https://aminet.net/package/util/cdity/PowerSnap22a [2] https://news.ycombinator.com/item?id=7631161 https://news.ycombinator.com/item?id=7631161
- 93po 1y ago
  thankfully most web browsing will be done by LLMs soon and that won't stop them, good riddance to the mess of a web that google has created
  - jgalt212 1y ago
    dead Internet for realz

hengheng 1y ago

I've also learned the hard way to Google "AM4 main board tier list" before buying. Some boards can run a 5950X in name only, while others can comfortably run it close to double its spec power all day. VRMs are a real differentiator for this tier of hardware. (If anyone can comment on the airflow required for 400-500W Epyc CPUs with the tiny VRM heatsinks that Supermicro uses, I'm all ears.)

ericdotlee 1y ago

Any reason you wouldn't opt for the 4090 or 5090?

Instantix 1y ago
3090 second hand can be found at something like $600.

yjftsjthsd-h 1y ago

> 3) The shopping link for the mainboard leads to the "ASUS ROG Strix X670E-E Gaming" model. This model can use the 2nd PCIe 5.0 port at only x4 speeds. The RTX 3090 can only do PCIe 4.0 of course so it will run at PCIe 4.0 x4. If you choose a desktop mainboard for having two GPUs, make sure it can run at PCIe x8 speeds when using both GPU slots! Having NVLink between the GPUs is not a replacement for having a fast connection between the CPU+RAM and the GPU and its VRAM. Forgive a noob question: I thought the connection to the GPU was actually fairly unimportant once the model was loaded, because sending input to the model and getting a response is low bandwidth? So it might matter if you're changing models a lot or doing a model that can work on video, but otherwise I thought it didn't really matter.

Tepix 1y ago
In general, if all you do is inference with a model that’s in VRAM, you’re right. OTOH it’s simply a matter of picking the right mainboard. If you have one of those sweet new MoE models that won‘t completely fit in your VRAM, offloading means you want PCIe bandwidth, because it will be a bottleneck. Also swapping between LLMs will be faster.