3 ms·

Comparing to Linux? No. There are no public numbers. However, I'm up to serving 709Gb/s of TLS encrypted traffic to from a single host to real Netflix custom

Comparing to Linux? No. There are no public numbers. However, I'm up to serving 709Gb/s of TLS encrypted traffic to from a single host to real Netflix customers with FreeBSD.

ksec 5y ago
Is that with CX7 and Dual Socket AMD CPU you mentioned awhile ago? ( I remember it was lost in delivery or some supply chain issues ) Or is that Single CPU? Because I thought you try to avoid the complexity of NUMA. Which I remember it was causing issues with first gen Zen.
- drewg123 5y ago
  4x CX6-DX in a dual-socket Rome. Its mostly an experimental platform, never intended for production, to shake out issues that we'll see when next generation platforms show up with DDR5 and PCIe Gen5. With a current single-socket setup we'd be limited by memory bandwidth around 500Gb/s or so, but 800Gb/s should hopefully be within reach from a single socket with DDR5 based servers. Especially since the limiting factor now is the traffic on the xGMI links between the sockets is unbalanced.
Melatonic 5y ago
Have you guys done a ton of your own customization and modification to the FreeBSD stack? Basically just wondering if both of you are correct and the default implementation of FreeBSD is in fact losing to other flavors of Linux but the networking customized OS' (not even just thinking of Netflix at this point) are still superior.
- drewg123 5y ago
  Most of our customizations and modifications have been upstreamed. We get similar performance on production 100g and 200g boxes when running the upstream kernel. I see no reason I shouldn't be able to hit 380Gb/s on a single-socket Rome box with an upstream kernel. I just haven't tried yet. Most of the changes that I have for the 700g number are changes to implement Disk centric NUMA siloing, which I would never upstream at this point because they are a pile of hacks. They are needed in order to change the NUMA node where memory is DMAed, so as to better utilize the xGMI links between AMD CPUs.
- ksec 5y ago
  You can search FreeBSD and Netflix on HN. They have presented their finding over the years with detailed comments in those threads.

Comparing to Linux? No. There are no public numbers. However, I'm up to serving 709Gb/s of TLS encrypted traffic to from a single host to real Netflix customers with FreeBSD.

ksec 5y ago

Is that with CX7 and Dual Socket AMD CPU you mentioned awhile ago? ( I remember it was lost in delivery or some supply chain issues ) Or is that Single CPU? Because I thought you try to avoid the complexity of NUMA. Which I remember it was causing issues with first gen Zen.

drewg123 5y ago
4x CX6-DX in a dual-socket Rome. Its mostly an experimental platform, never intended for production, to shake out issues that we'll see when next generation platforms show up with DDR5 and PCIe Gen5. With a current single-socket setup we'd be limited by memory bandwidth around 500Gb/s or so, but 800Gb/s should hopefully be within reach from a single socket with DDR5 based servers. Especially since the limiting factor now is the traffic on the xGMI links between the sockets is unbalanced.

Melatonic 5y ago

Have you guys done a ton of your own customization and modification to the FreeBSD stack? Basically just wondering if both of you are correct and the default implementation of FreeBSD is in fact losing to other flavors of Linux but the networking customized OS' (not even just thinking of Netflix at this point) are still superior.

drewg123 5y ago
Most of our customizations and modifications have been upstreamed. We get similar performance on production 100g and 200g boxes when running the upstream kernel. I see no reason I shouldn't be able to hit 380Gb/s on a single-socket Rome box with an upstream kernel. I just haven't tried yet. Most of the changes that I have for the 700g number are changes to implement Disk centric NUMA siloing, which I would never upstream at this point because they are a pile of hacks. They are needed in order to change the NUMA node where memory is DMAed, so as to better utilize the xGMI links between AMD CPUs.
ksec 5y ago
You can search FreeBSD and Netflix on HN. They have presented their finding over the years with detailed comments in those threads.