4 ms·
The lanes come from the CPU you put into the board, not the board itself. Although yes they are still cheap, at least if you go with something like the EPYC 725
by kllrnohj 7y ago
The lanes come from the CPU you put into the board, not the board itself. Although yes they are still cheap, at least if you go with something like the EPYC 7252 at ~$500 (which still has the full 128 PCI-E 4.0 lanes)
That said I have no idea how you would actually feed that many PCI-E lanes with an EPYC 7252, but if you can pull it off it's an insane $/lane value.
- chx 7y agoI know it's the CPU but the board I linked is a very rare standard ATX board, almost all boards are proprietary. I presume you could build an insane fast fileserver with a real lot of M.2 disks and multiple 100GbE ports?
- Robotbeat 7y agoThat's right. You can also use a PCIe expansion chassis (there are already ones supporting PCIe 4.0), giving you plenty of space for dual-slot-width cards.
- kllrnohj 7y agoSure but I don't think the 8 core epyc would actually keep up with that many NVME drives. At least not if you tried to actually hit 24+ of them at once. Linus tech tips tried this and had to upgrade the CPU from the 24 core epyc to the 32 core to get performance up to what they wanted. https://youtu.be/xWjOh0Ph8uM https://youtu.be/xWjOh0Ph8uM Maybe just a bad deployment but there is overhead in filesystems. Especially with checksums and compression and redundancy and etc...
- Robotbeat 7y agoIt's possible to bypass the CPU in some cases using NVMe over an RDMA layer with Infiniband. PCIe 4.0 dual-port 200Gbps Infiniband/Ethernet adapters exist[1] which are compatible with this approach: https://store.mellanox.com/products/mellanox-mcx653106a-hdat-sp-single-pack-connectx-6-vpi-adapter-card-hdr-ib-and-200gbe-dual-port-qsfp56-pcie4-0-x16-tall-bracket.html https://store.mellanox.com/products/mellanox-mcx653106a-hdat... [1]Although you can't saturate both of them through even a 16 lane PCIe 4.0 port which has ~250Gbps of throughput each way.... Which to me means that PCIe 4.0 is not at all too soon.
- chx 7y agoAlso if you calculate the USD per 3.0 lane value you will find you can go much,much higher in CPU prices. If you look at various combos you will find it very rare for the server CPU+board price divided by the number of 3.0 lanes or equivalent to be below 10USD.