5 ms·

One related issue is the death of integrated PCIe switches on motherboards. High-end boards used to have PCIe hubs on them so you could plugin in a lot of cards

One related issue is the death of integrated PCIe switches on motherboards. High-end boards used to have PCIe hubs on them so you could plugin in a lot of cards and still get all the bandwidth out of a 16-lane CPU. The reason why they don't do this is because PLX got bought out by Broadcom and they jacked up the price of the chips. It turns out their most lucrative market was storage, because datacenters want to connect a lot of high-speed NVMe to one machine and they will pay buckets for a chip that will let them all share bandwidth. So now consumers that want a lot of devices need a CPU with a lot of lanes, which meant buying into a more expensive high-end desktop platform with more than the standard of 16 or 20 lanes on it. Except these are also becoming scarce; both Intel and AMD's high-end desktop platforms haven't been updated in years. The way high-end used to work was that they were just last-gen server chips with overclocking and more consumer-oriented boards, so they got the extra lanes and memory capacity, but you still got the gobs of USB and so on that you'd see on a consumer board. So with no lane expanders on motherboards anymore, and no high-end platforms with higher lane count CPUs in them, the only other option for buying something with lots of slots in it is to buy server gear. Not only is that more expensive, but you also lose out on all the consumer-grade creature comforts you expect[0] and have to deal with the added noise of small fans in a rackmount case. [0] VGA is considered standard video output on most server motherboards

arminiusreturns 4y ago
Can confirm, much of my high-compute architecture design work centered around heavy analysis and comparison of existing and promised backplanes. Especially when you throw images in the mix and need gpus, but you need the whole stack to be HC/HT or you hit bottlenecks later.
PragmaticPulp 4y ago
> One related issue is the death of integrated PCIe switches on motherboards. High-end boards used to have PCIe hubs on them so you could plugin in a lot of cards and still get all the bandwidth out of a 16-lane CPU. PCIe switches weren't common or necessary for most consumer applications. Workstation and server CPUs have plenty of lanes on their own. PCIe switches occupied a relatively narrow use case in the middle where people wanted to use consumer CPUs but attach a lot of add-in cards and share bandwidth across them. Even that use case has been eroded a bit with PCIe splitting options, where an x16 lane can be split into x8/x8 or even x4/x4/x4/x4 depending on the motherboard. An x4 PCIe 4.0 lane is as fast as an old x16 PCIe 2.0 lane, so that's a significant amount of bandwidth still. PCIe 5.0 takes this even further, where an x16 PCIe 5.0 slot has over 60GB/sec of bandwidth, which is a lot even when divided into x4 or x2 channels. AMD especially made platforms with a lot of PCIe lanes relatively affordable. An AMD workstation motherboard and CPU with a lot of PCIe lanes might not actually be that much more expensive than a hypothetical consumer CPU with a PCIe switch. The other issue is that with a PCIe switch is that you're still limited to upstream bandwidth, so if you need x16 bandwidth running to multiple cards at the same time, a PCIe switch doesn't actually help you.
- bilegeek 4y ago
  It's still a problem if you want to run previous gen hardware. For instance, if I have a Gen4 motherboard and pass through a second GPU to a VM - so the second GPU at Gen3 x8 plus the host GPU at Gen4 x8 - I'm only using a quarter instead of half the throughput the hardware actually supports, because Gen3 is half the speed of Gen4 per lane. Or even think about Gen5, doubling Gen4 speeds: With a PCIe switch, I could get essentially 3/4 performance out of a Gen5 host GPU, and full performance out of a Gen3 second, instead of 1/2 and 1/4 respectively.
  - Dalewyn 4y ago
    Do GPUs even saturate PCIe3 x16 or x8 connections, let alone PCIe4 or 5, that we need to be worried about bottlenecks? The only thing I'm aware of that will actually saturate a PCIe connection is NVME storage.
    - KennyBlanken 4y ago
      For gaming, not even close. I've often had a GPU monitor running, charted bus utilization...and it's barely a few percent no matter what game I'm running, except for a very brief spike during texture and model load-in at the start of a round/level.
      - bilegeek 4y ago
        Specifically for VM's, something like Looking Glass (shared framebuffer so you don't need a dedicated monitor) needs the extra bandwidth to get the video signal through without tanking performance, especially at higher resolutions and refresh rates. Aside from that, I agree it's not too big an issue for 95% of computer users, but it's something to keep in mind. [1]https://looking-glass.io/ https://looking-glass.io/
        wtallis 4y ago
        To quantify this: a 4k 60Hz uncompressed video stream needs approximately one lane of PCIe gen4 (though only in one direction, while PCIe has the bandwidth in both directions). So any realistically obtainable resolution and refresh rate will not be a problem for sharing the framebuffer on a GPU with PCIe gen4 x16, but lower-end GPUs often have just x8 and there are even some mobile GPUs with x4 being packaged onto desktop graphics cards now.
thequux 4y ago
I'm running a server motherboard (Supermicro H12DSi-N6), and while the processing power is wonderful, the tradeoffs were a bit surprising: * I can't disable the BMC video card, which means that when I'm running Windows, there's a 800x600 monitor that will never go away and windows occasionally decide to open on it. This is solvable by opening the BMC on the machine itself and using the IKVM webapp to drag it onto my main screen; this takes a very careful flick of the mouse. * There's no built-in audio; in order to listen to Spotify, I use a USB audio card. * The board only has 6 USB ports, only 2 of which are USB-3. Thus, I get to use a lot of hubs. * It takes a long time to boot past the firmware; generally 3-4 minutes to warm boot and 6-8 cold. * While it can be used as a single-socket board (which I do, because I don't have the funds for a second CPU just yet), I lose a bunch of PCIe slots doing so; half the PCIe lanes for the first CPU are direct wired to the second socket for the inter-CPU link.
- sdrinf 4y ago
  | * It takes a long time to boot past the firmware; generally 3-4 minutes to warm boot and 6-8 cold. ^^ why is that? is it just the firmware being buggy, or is there any legit reasons why they need to self-check _for minutes_?
  - zeta0134 4y ago
    Server hardware operates remote and headless, so it performs much more exhaustive memory checks and will try to proactively report failures. Good datacenters will catch these and open a ticket to replace failing hardware. Beyond that, lots of servers will then enumerate all of the network interfaces and try to initiate a net boot, and only then fall back to the hard drive. Some tweaking in the bios can usually disable these features and gain a bunch of time, but again in a headless remote context this default behavior is incredibly useful for recovery purposes. We like to avoid actually running out with a physical crash cart if we can help it :)
- justinlloyd 4y ago
  Bridge jumper JPVGA1 directly on the motherboard and it will disable the onboard VGA. Switch BMC priority from onboard (default) to offboard, I don't think this is necessary if you bridge JPVGA1. Get yourself a nice Highpoint USB 3.0 card with quad controllers on each port. 20Gbps of bandwidth in aggregate. I believe there is a jumper on the motherboard that will bridge the missing PCIe slots in single CPU configuration.
tanelpoder 4y ago
AMD definitely has a recent high end desktop/workstation platform with its ThreadRipper Pro WX CPUs (EPYC server CPUs) that have 128 PCIe 4.0 lanes and 8 memory channels. And enough PCIe slots to actually use this throughput with the right kind of hardware, like a bunnch of PCIe4.0 NVMe SSDs [1]. Yep, the platform is 2 years old, but supports both the Zen2 and current Zen3 CPUs (but not PCIe5.0 & DDR5 as Zen 4 requires a different CPU socket and is only announced right now, not available from AMD until the end this month). However, 8 x DDR4 channels and 128 x PCIe4.0 lanes still give you better total bandwidth than 2 x DDR5 and 24 x PCIe5.0 like the few currently available DDR5/PCIe5 platforms support. [1] https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation/ https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-th...
- bick_nyers 4y ago
  The unfortunate part is that Threadripper nowadays is OEM only, thus it is very expensive. You can just switch to EPYC with hardly any difference in price point.
  - tanelpoder 4y ago
    The Threadripper Pro 5000-series is available for retail purchases [1]. When 3000-series was launched, it was first OEM only (Lenovo) but later became available in retail too. [1] https://www.amazon.com/AMD-ThreadripperTM-PRO-5975WX-64-Thread/dp/B0B5VH1WPC https://www.amazon.com/AMD-ThreadripperTM-PRO-5975WX-64-Thre...
    - bick_nyers 4y ago
      Oh nice, I must have missed this. Motherboard prices are leaving something to be desired (lowest price is $900), but still, this is good news. Edit: On second glance, the markup compared to last-gen Threadripper is gnarly. 24 core SKU MSRP is $2.4k, compared to the 3000 series SKU which MSRP for $1.4k. At that point just go EPYC.
chrisseaton 4y ago
How do all these lanes get into the CPU? There are only so many pins on a chip. Are they time-shared? If so why do you need n physical slots and not just a switched network that lets you daisy chain as many as you can switch? (I’m an architecture lay-person.)
- davrosthedalek 4y ago
  Threadripper comes in a seriously big package. 4094 pins, compared to 1718 for AM5 (the new ryzen socket).
- wtallis 4y ago
  PCIe already is a packet-switched network, but the big and fast switches are very expensive.
tim-- 4y ago
AMD Threadripper got an update this year, and a new release should come early next year. 128 PCIe lanes.
nyanpasu64 4y ago
I'm raise you no working video output on my consumer motherboard! cries in GPU-less AMD CPU
hinkley 4y ago
And other creature comforts like decibels, I suspect.

arminiusreturns 4y ago

Can confirm, much of my high-compute architecture design work centered around heavy analysis and comparison of existing and promised backplanes. Especially when you throw images in the mix and need gpus, but you need the whole stack to be HC/HT or you hit bottlenecks later.

PragmaticPulp 4y ago

> One related issue is the death of integrated PCIe switches on motherboards. High-end boards used to have PCIe hubs on them so you could plugin in a lot of cards and still get all the bandwidth out of a 16-lane CPU. PCIe switches weren't common or necessary for most consumer applications. Workstation and server CPUs have plenty of lanes on their own. PCIe switches occupied a relatively narrow use case in the middle where people wanted to use consumer CPUs but attach a lot of add-in cards and share bandwidth across them. Even that use case has been eroded a bit with PCIe splitting options, where an x16 lane can be split into x8/x8 or even x4/x4/x4/x4 depending on the motherboard. An x4 PCIe 4.0 lane is as fast as an old x16 PCIe 2.0 lane, so that's a significant amount of bandwidth still. PCIe 5.0 takes this even further, where an x16 PCIe 5.0 slot has over 60GB/sec of bandwidth, which is a lot even when divided into x4 or x2 channels. AMD especially made platforms with a lot of PCIe lanes relatively affordable. An AMD workstation motherboard and CPU with a lot of PCIe lanes might not actually be that much more expensive than a hypothetical consumer CPU with a PCIe switch. The other issue is that with a PCIe switch is that you're still limited to upstream bandwidth, so if you need x16 bandwidth running to multiple cards at the same time, a PCIe switch doesn't actually help you.

bilegeek 4y ago
It's still a problem if you want to run previous gen hardware. For instance, if I have a Gen4 motherboard and pass through a second GPU to a VM - so the second GPU at Gen3 x8 plus the host GPU at Gen4 x8 - I'm only using a quarter instead of half the throughput the hardware actually supports, because Gen3 is half the speed of Gen4 per lane. Or even think about Gen5, doubling Gen4 speeds: With a PCIe switch, I could get essentially 3/4 performance out of a Gen5 host GPU, and full performance out of a Gen3 second, instead of 1/2 and 1/4 respectively.
- Dalewyn 4y ago
  Do GPUs even saturate PCIe3 x16 or x8 connections, let alone PCIe4 or 5, that we need to be worried about bottlenecks? The only thing I'm aware of that will actually saturate a PCIe connection is NVME storage.
  - KennyBlanken 4y ago
    For gaming, not even close. I've often had a GPU monitor running, charted bus utilization...and it's barely a few percent no matter what game I'm running, except for a very brief spike during texture and model load-in at the start of a round/level.
    - bilegeek 4y ago
      Specifically for VM's, something like Looking Glass (shared framebuffer so you don't need a dedicated monitor) needs the extra bandwidth to get the video signal through without tanking performance, especially at higher resolutions and refresh rates. Aside from that, I agree it's not too big an issue for 95% of computer users, but it's something to keep in mind. [1]https://looking-glass.io/ https://looking-glass.io/
      - wtallis 4y ago
        To quantify this: a 4k 60Hz uncompressed video stream needs approximately one lane of PCIe gen4 (though only in one direction, while PCIe has the bandwidth in both directions). So any realistically obtainable resolution and refresh rate will not be a problem for sharing the framebuffer on a GPU with PCIe gen4 x16, but lower-end GPUs often have just x8 and there are even some mobile GPUs with x4 being packaged onto desktop graphics cards now.

thequux 4y ago

I'm running a server motherboard (Supermicro H12DSi-N6), and while the processing power is wonderful, the tradeoffs were a bit surprising: * I can't disable the BMC video card, which means that when I'm running Windows, there's a 800x600 monitor that will never go away and windows occasionally decide to open on it. This is solvable by opening the BMC on the machine itself and using the IKVM webapp to drag it onto my main screen; this takes a very careful flick of the mouse. * There's no built-in audio; in order to listen to Spotify, I use a USB audio card. * The board only has 6 USB ports, only 2 of which are USB-3. Thus, I get to use a lot of hubs. * It takes a long time to boot past the firmware; generally 3-4 minutes to warm boot and 6-8 cold. * While it can be used as a single-socket board (which I do, because I don't have the funds for a second CPU just yet), I lose a bunch of PCIe slots doing so; half the PCIe lanes for the first CPU are direct wired to the second socket for the inter-CPU link.

sdrinf 4y ago
| * It takes a long time to boot past the firmware; generally 3-4 minutes to warm boot and 6-8 cold. ^^ why is that? is it just the firmware being buggy, or is there any legit reasons why they need to self-check _for minutes_?
- zeta0134 4y ago
  Server hardware operates remote and headless, so it performs much more exhaustive memory checks and will try to proactively report failures. Good datacenters will catch these and open a ticket to replace failing hardware. Beyond that, lots of servers will then enumerate all of the network interfaces and try to initiate a net boot, and only then fall back to the hard drive. Some tweaking in the bios can usually disable these features and gain a bunch of time, but again in a headless remote context this default behavior is incredibly useful for recovery purposes. We like to avoid actually running out with a physical crash cart if we can help it :)
justinlloyd 4y ago
Bridge jumper JPVGA1 directly on the motherboard and it will disable the onboard VGA. Switch BMC priority from onboard (default) to offboard, I don't think this is necessary if you bridge JPVGA1. Get yourself a nice Highpoint USB 3.0 card with quad controllers on each port. 20Gbps of bandwidth in aggregate. I believe there is a jumper on the motherboard that will bridge the missing PCIe slots in single CPU configuration.

tanelpoder 4y ago

AMD definitely has a recent high end desktop/workstation platform with its ThreadRipper Pro WX CPUs (EPYC server CPUs) that have 128 PCIe 4.0 lanes and 8 memory channels. And enough PCIe slots to actually use this throughput with the right kind of hardware, like a bunnch of PCIe4.0 NVMe SSDs [1]. Yep, the platform is 2 years old, but supports both the Zen2 and current Zen3 CPUs (but not PCIe5.0 & DDR5 as Zen 4 requires a different CPU socket and is only announced right now, not available from AMD until the end this month). However, 8 x DDR4 channels and 128 x PCIe4.0 lanes still give you better total bandwidth than 2 x DDR5 and 24 x PCIe5.0 like the few currently available DDR5/PCIe5 platforms support. [1] https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation/ https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-th...

bick_nyers 4y ago
The unfortunate part is that Threadripper nowadays is OEM only, thus it is very expensive. You can just switch to EPYC with hardly any difference in price point.
- tanelpoder 4y ago
  The Threadripper Pro 5000-series is available for retail purchases [1]. When 3000-series was launched, it was first OEM only (Lenovo) but later became available in retail too. [1] https://www.amazon.com/AMD-ThreadripperTM-PRO-5975WX-64-Thread/dp/B0B5VH1WPC https://www.amazon.com/AMD-ThreadripperTM-PRO-5975WX-64-Thre...
  - bick_nyers 4y ago
    Oh nice, I must have missed this. Motherboard prices are leaving something to be desired (lowest price is $900), but still, this is good news. Edit: On second glance, the markup compared to last-gen Threadripper is gnarly. 24 core SKU MSRP is $2.4k, compared to the 3000 series SKU which MSRP for $1.4k. At that point just go EPYC.

chrisseaton 4y ago

How do all these lanes get into the CPU? There are only so many pins on a chip. Are they time-shared? If so why do you need n physical slots and not just a switched network that lets you daisy chain as many as you can switch? (I’m an architecture lay-person.)

davrosthedalek 4y ago
Threadripper comes in a seriously big package. 4094 pins, compared to 1718 for AM5 (the new ryzen socket).
wtallis 4y ago
PCIe already is a packet-switched network, but the big and fast switches are very expensive.

tim-- 4y ago

AMD Threadripper got an update this year, and a new release should come early next year. 128 PCIe lanes.

nyanpasu64 4y ago

I'm raise you no working video output on my consumer motherboard! cries in GPU-less AMD CPU

hinkley 4y ago

And other creature comforts like decibels, I suspect.