4 ms·

Yea, the platform cost is higher. I ended up making a build you could make for ~$1,500 now. It was ~$1,600 when I made it. The biggest feature I'm interested in

Yea, the platform cost is higher. I ended up making a build you could make for ~$1,500 now. It was ~$1,600 when I made it. The biggest feature I'm interested in is the availability of PCIe lanes. I want this for adding a 10G nic later and two GPUs at some point as well (host & guest). If you don't need those features you're completely correct about the 3950x.

jchw 7y ago
Ah, GPU passthrough? My biggest problem with virtualization is USB. I have a libvirt with GPU passthrough setup that works great, but have been unable to get a USB controller of any sort to passthrough; always winds up in a group with a bunch of other PCI-e devices. And ordinary forwarding with SPICE or something isn’t really sufficient for what I’d like to set up...
- voltagex_ 7y ago
  Have you tried IOMMU splitting? https://forums.unraid.net/topic/72027-iommu-group-splitting-for-pcie-passthrough-no-success-please-help/ https://forums.unraid.net/topic/72027-iommu-group-splitting-... (disclaimer - I don't own a board that can do this, I will one day, though).
  - jchw 7y ago
    I have, though it causes a lot of stability problems :(
- simcop2387 7y ago
  It's technically a security risk, but take a look at the acs patch that's out there. It'll forcefully split up the iommu groups to the hypervisor so you can do the pass through, but it does mean that the cards that were in the same group before can technically see each other's dma and other stuff on the bus. For anything other than a shared host it's pretty much fine though.
floatboth 7y ago
This should be doable on desktop Ryzen. I currently have one GPU (x16) + NVMe SSD (x4) + 10G NIC (x4 from chipset). The 16x can be split x8/x8 for dual GPU. https://linustechtips.com/main/topic/799836-pcie-lanes-for-ryzen/?tab=comments#comment-10061434 https://linustechtips.com/main/topic/799836-pcie-lanes-for-r... — nice diagram of lanes
- gravypod 7y ago
  If you're running 10G through the chipset and running GPUs on a x8 config you're going to bottleneck yourself in my opinion.
  - AstralStorm 7y ago
    Except this is pcie 4 vs pcie 3 so it's double the lanes for the latter, so no, x8 pcie 4 will not bottleneck anything. Unfortunately GPU do not support pcie 4 yet, but it's not a problem for integrated 10GbE.
    - gravypod 7y ago
      I thought AMD said their. New card was pcie 4
      - floatboth 7y ago
        Yes, the Navi card of course supports gen4, and even the Vega20 did too. At least in the original Instinct variant (most places on the internet say that gen4 was cut on the consumer Radeon VII card)
  - floatboth 7y ago
    My single port Intel X520 achieves line rate through x4 chipset lanes just fine. GPUs generally don't come close to saturating x8 3.0 lanes, unless you have a very specific workload (like the new 3dmark bandwidth benchmark AMD used to demo PCIe 4.0). Games don't do nearly enough asset streaming to use a lot of bandwidth, since the amount of assets used at the same time is limited by VRAM size, and most stuff is kept around for quite some time. Offline 3D renderers like Blender Cycles IIRC just upload the whole scene at once and then path tracing happens in VRAM without much I/O. For buttcoin mining, people literally use boards with tons of x1 slots + risers. No idea how neural nets behave, but would make sense that they also just keep updating the weights in VRAM.

jchw 7y ago

Ah, GPU passthrough? My biggest problem with virtualization is USB. I have a libvirt with GPU passthrough setup that works great, but have been unable to get a USB controller of any sort to passthrough; always winds up in a group with a bunch of other PCI-e devices. And ordinary forwarding with SPICE or something isn’t really sufficient for what I’d like to set up...

voltagex_ 7y ago
Have you tried IOMMU splitting? https://forums.unraid.net/topic/72027-iommu-group-splitting-for-pcie-passthrough-no-success-please-help/ https://forums.unraid.net/topic/72027-iommu-group-splitting-... (disclaimer - I don't own a board that can do this, I will one day, though).
- jchw 7y ago
  I have, though it causes a lot of stability problems :(
simcop2387 7y ago
It's technically a security risk, but take a look at the acs patch that's out there. It'll forcefully split up the iommu groups to the hypervisor so you can do the pass through, but it does mean that the cards that were in the same group before can technically see each other's dma and other stuff on the bus. For anything other than a shared host it's pretty much fine though.

floatboth 7y ago

This should be doable on desktop Ryzen. I currently have one GPU (x16) + NVMe SSD (x4) + 10G NIC (x4 from chipset). The 16x can be split x8/x8 for dual GPU. https://linustechtips.com/main/topic/799836-pcie-lanes-for-ryzen/?tab=comments#comment-10061434 https://linustechtips.com/main/topic/799836-pcie-lanes-for-r... — nice diagram of lanes

gravypod 7y ago
If you're running 10G through the chipset and running GPUs on a x8 config you're going to bottleneck yourself in my opinion.
- AstralStorm 7y ago
  Except this is pcie 4 vs pcie 3 so it's double the lanes for the latter, so no, x8 pcie 4 will not bottleneck anything. Unfortunately GPU do not support pcie 4 yet, but it's not a problem for integrated 10GbE.
  - gravypod 7y ago
    I thought AMD said their. New card was pcie 4
    - floatboth 7y ago
      Yes, the Navi card of course supports gen4, and even the Vega20 did too. At least in the original Instinct variant (most places on the internet say that gen4 was cut on the consumer Radeon VII card)
- floatboth 7y ago
  My single port Intel X520 achieves line rate through x4 chipset lanes just fine. GPUs generally don't come close to saturating x8 3.0 lanes, unless you have a very specific workload (like the new 3dmark bandwidth benchmark AMD used to demo PCIe 4.0). Games don't do nearly enough asset streaming to use a lot of bandwidth, since the amount of assets used at the same time is limited by VRAM size, and most stuff is kept around for quite some time. Offline 3D renderers like Blender Cycles IIRC just upload the whole scene at once and then path tracing happens in VRAM without much I/O. For buttcoin mining, people literally use boards with tons of x1 slots + risers. No idea how neural nets behave, but would make sense that they also just keep updating the weights in VRAM.