5 ms·

71 GB/s is 568 Gbit/s so you'll need about 3 dual 100 Gbit/s cards to pump data out at the rate you can read it from the NVMe drives. And ethernet (unless LAN

by guerby 6y ago

71 GB/s is 568 Gbit/s so you'll need about 3 dual 100 Gbit/s cards to pump data out at the rate you can read it from the NVMe drives. And ethernet (unless LAN jumbo frames) is about 1.5kByte per frame (not 4kB). One such PC should be able to do 100k simultaneous 5 Mbps HD streams. Testing this would be fun :)

tanelpoder 6y ago
I should (finally) receive my RTX 3090 card today (PCIe 4.0 too!), I guess here goes my weekend (and the following weekends over a couple of years)!
zamadatix 6y ago
Mellanox has a 2x200 Gbps NIC these days. Haven't gotten to play with it yet though.
- tanelpoder 6y ago
  Which NICs would you recommend for me to buy for testing at least 1x100 Gbps (ideally 200 Gbps?) networking between this machine (PCIe 4.0) and an Intel Xeon one that I have with PCIe 3.0. Don't want to spend much money, so the cards don't need to be too enterprisey, just fast. And - do such cards even allow direct "cross" connection without a switch in between?
  - drewg123 6y ago
    All 100G is enterprisy. For a cheap solution, I'd get a pair of used Mellanox ConnectX4 or Chelsio T6, and a QSFP28 direct attach copper cable.
  - zamadatix 6y ago
    +1 on what the sibling comment said. As for directly connecting them absolutely, works great. Id recommend a cheap DAC off fs.com to connect them in that case.
  - namibj 6y ago
    If you care about price, check out (used, ofc) infiniband cards. They all seem to offer/suggest daisy-chain connectivity at least for those with two ports per card as one potential topology.
drewg123 6y ago
At Netflix, I'm playing with an EPYC 7502P with 16 NVME and dual 2x100 Mellanox ConnectX6-DX NICs. With hardware kTLS offload, we're able to serve about 350Gb/s of real customer traffic. This goes down to about 240Gb/s when using software kTLS, due to memory bandwidth limits. This is all FreeBSD, and is the evolution of the work described in my talk at the last EuroBSDCon in 2019: https://papers.freebsd.org/2019/eurobsdcon/gallatin-numa_optimizations_in_the_freebsd_network_stack/ https://papers.freebsd.org/2019/eurobsdcon/gallatin-numa_opt...
- ksec 6y ago
  >we're able to serve about 350Gb/s of real customer traffic. I still remember the post about breaking 100Gbps barrier, that was may be in 2016 or 17 ? And wasn't that long ago it was 200Gbps and if I remember correct it was hitting memory bandwidth barrier as well. And now 350Gbps?! So what's next? Wait for DDR5? Or moving to some memory controller black magic like POWER10?
  - drewg123 6y ago
    Yes, before hardware inline kTLS offload, we were limited to 200Gb/s or so with Naples. With Rome, its a bit higher. But hardware inline kTLS with the Mellanox CX6-DX eliminates memory bandwidth as a bottleneck. The current bottleneck is IO related, and its unclear what the issue is. We're working with the hardware vendors to try to figure it out. We should be getting about 390Gb/s
    - ksec 6y ago
      Oh wow! Cant wait to hear more about it.
    - jiggawatts 6y ago
      > But hardware inline kTLS with the Mellanox CX6-DX eliminates memory bandwidth as a bottleneck. For a while now I had operated under the assumption that CPU-based crypto with AES-GCM was faster than most hardware offload cards. What makes the Mellanox NIC perform better? I.e.: Why does memory bandwidth matter to TLS? Aren't you encrypting data "on the fly", while it is still resident in the CPU caches? > We're working with the hardware vendors to try to figure it out. We should be getting about 390Gb/s Something I explained to a colleague recently is that a modern CPU gains or loses more computer power from a 1° C temperature difference in the room's air than my first four computers had combined. You're basically complaining that you're unable to get a mere 10% of the expected throughput. But put in absolute terms, that's 40 Gbps, which is about 10x more than what a typical server in 2020 can put out on the network. (Just because you have 10 Gbps NICs doesn't mean you can get 10 Gbps! Try iperf3 and you'll be shocked that you're lucky if you can crack 5 Gbps in practice)

by guerby 6y ago

tanelpoder 6y ago

I should (finally) receive my RTX 3090 card today (PCIe 4.0 too!), I guess here goes my weekend (and the following weekends over a couple of years)!

zamadatix 6y ago

Mellanox has a 2x200 Gbps NIC these days. Haven't gotten to play with it yet though.

tanelpoder 6y ago
Which NICs would you recommend for me to buy for testing at least 1x100 Gbps (ideally 200 Gbps?) networking between this machine (PCIe 4.0) and an Intel Xeon one that I have with PCIe 3.0. Don't want to spend much money, so the cards don't need to be too enterprisey, just fast. And - do such cards even allow direct "cross" connection without a switch in between?
- drewg123 6y ago
  All 100G is enterprisy. For a cheap solution, I'd get a pair of used Mellanox ConnectX4 or Chelsio T6, and a QSFP28 direct attach copper cable.
- zamadatix 6y ago
  +1 on what the sibling comment said. As for directly connecting them absolutely, works great. Id recommend a cheap DAC off fs.com to connect them in that case.
- namibj 6y ago
  If you care about price, check out (used, ofc) infiniband cards. They all seem to offer/suggest daisy-chain connectivity at least for those with two ports per card as one potential topology.

drewg123 6y ago

At Netflix, I'm playing with an EPYC 7502P with 16 NVME and dual 2x100 Mellanox ConnectX6-DX NICs. With hardware kTLS offload, we're able to serve about 350Gb/s of real customer traffic. This goes down to about 240Gb/s when using software kTLS, due to memory bandwidth limits. This is all FreeBSD, and is the evolution of the work described in my talk at the last EuroBSDCon in 2019: https://papers.freebsd.org/2019/eurobsdcon/gallatin-numa_optimizations_in_the_freebsd_network_stack/ https://papers.freebsd.org/2019/eurobsdcon/gallatin-numa_opt...

ksec 6y ago
>we're able to serve about 350Gb/s of real customer traffic. I still remember the post about breaking 100Gbps barrier, that was may be in 2016 or 17 ? And wasn't that long ago it was 200Gbps and if I remember correct it was hitting memory bandwidth barrier as well. And now 350Gbps?! So what's next? Wait for DDR5? Or moving to some memory controller black magic like POWER10?
- drewg123 6y ago
  Yes, before hardware inline kTLS offload, we were limited to 200Gb/s or so with Naples. With Rome, its a bit higher. But hardware inline kTLS with the Mellanox CX6-DX eliminates memory bandwidth as a bottleneck. The current bottleneck is IO related, and its unclear what the issue is. We're working with the hardware vendors to try to figure it out. We should be getting about 390Gb/s
  - ksec 6y ago
    Oh wow! Cant wait to hear more about it.
  - jiggawatts 6y ago
    > But hardware inline kTLS with the Mellanox CX6-DX eliminates memory bandwidth as a bottleneck. For a while now I had operated under the assumption that CPU-based crypto with AES-GCM was faster than most hardware offload cards. What makes the Mellanox NIC perform better? I.e.: Why does memory bandwidth matter to TLS? Aren't you encrypting data "on the fly", while it is still resident in the CPU caches? > We're working with the hardware vendors to try to figure it out. We should be getting about 390Gb/s Something I explained to a colleague recently is that a modern CPU gains or loses more computer power from a 1° C temperature difference in the room's air than my first four computers had combined. You're basically complaining that you're unable to get a mere 10% of the expected throughput. But put in absolute terms, that's 40 Gbps, which is about 10x more than what a typical server in 2020 can put out on the network. (Just because you have 10 Gbps NICs doesn't mean you can get 10 Gbps! Try iperf3 and you'll be shocked that you're lucky if you can crack 5 Gbps in practice)