4 ms·

It depends on your workload. For static content, especially kTLS encrypted static content, FreeBSD is quite a bit better.

frankjr 5y ago
Do you have any numbers you can share publicly? What's the reason it's better (Linux has in-kernel TLS as well, correct?)?. It would make for a great topic for Netflix TechBlog.
- doublerabbit 5y ago
  > At this point, we’re able to serve 100% TLS traffic comfortably at 90 Gbps using the default FreeBSD TCP stack. https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99?source=---------0----------------------- https://netflixtechblog.com/serving-100-gbps-from-an-open-co...
  - loeg 5y ago
    That was 2017 -- it's quite a bit higher now.
- drewg123 5y ago
  Comparing to Linux? No. There are no public numbers. However, I'm up to serving 709Gb/s of TLS encrypted traffic to from a single host to real Netflix customers with FreeBSD.
  - ksec 5y ago
    Is that with CX7 and Dual Socket AMD CPU you mentioned awhile ago? ( I remember it was lost in delivery or some supply chain issues ) Or is that Single CPU? Because I thought you try to avoid the complexity of NUMA. Which I remember it was causing issues with first gen Zen.
    - drewg123 5y ago
      4x CX6-DX in a dual-socket Rome. Its mostly an experimental platform, never intended for production, to shake out issues that we'll see when next generation platforms show up with DDR5 and PCIe Gen5. With a current single-socket setup we'd be limited by memory bandwidth around 500Gb/s or so, but 800Gb/s should hopefully be within reach from a single socket with DDR5 based servers. Especially since the limiting factor now is the traffic on the xGMI links between the sockets is unbalanced.
  - Melatonic 5y ago
    Have you guys done a ton of your own customization and modification to the FreeBSD stack? Basically just wondering if both of you are correct and the default implementation of FreeBSD is in fact losing to other flavors of Linux but the networking customized OS' (not even just thinking of Netflix at this point) are still superior.
    - drewg123 5y ago
      Most of our customizations and modifications have been upstreamed. We get similar performance on production 100g and 200g boxes when running the upstream kernel. I see no reason I shouldn't be able to hit 380Gb/s on a single-socket Rome box with an upstream kernel. I just haven't tried yet. Most of the changes that I have for the 700g number are changes to implement Disk centric NUMA siloing, which I would never upstream at this point because they are a pile of hacks. They are needed in order to change the NUMA node where memory is DMAed, so as to better utilize the xGMI links between AMD CPUs.
    - ksec 5y ago
      You can search FreeBSD and Netflix on HN. They have presented their finding over the years with detailed comments in those threads.

frankjr 5y ago

Do you have any numbers you can share publicly? What's the reason it's better (Linux has in-kernel TLS as well, correct?)?. It would make for a great topic for Netflix TechBlog.

doublerabbit 5y ago
> At this point, we’re able to serve 100% TLS traffic comfortably at 90 Gbps using the default FreeBSD TCP stack. https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99?source=---------0----------------------- https://netflixtechblog.com/serving-100-gbps-from-an-open-co...
- loeg 5y ago
  That was 2017 -- it's quite a bit higher now.
drewg123 5y ago
Comparing to Linux? No. There are no public numbers. However, I'm up to serving 709Gb/s of TLS encrypted traffic to from a single host to real Netflix customers with FreeBSD.
- ksec 5y ago
  Is that with CX7 and Dual Socket AMD CPU you mentioned awhile ago? ( I remember it was lost in delivery or some supply chain issues ) Or is that Single CPU? Because I thought you try to avoid the complexity of NUMA. Which I remember it was causing issues with first gen Zen.
  - drewg123 5y ago
    4x CX6-DX in a dual-socket Rome. Its mostly an experimental platform, never intended for production, to shake out issues that we'll see when next generation platforms show up with DDR5 and PCIe Gen5. With a current single-socket setup we'd be limited by memory bandwidth around 500Gb/s or so, but 800Gb/s should hopefully be within reach from a single socket with DDR5 based servers. Especially since the limiting factor now is the traffic on the xGMI links between the sockets is unbalanced.
- Melatonic 5y ago
  Have you guys done a ton of your own customization and modification to the FreeBSD stack? Basically just wondering if both of you are correct and the default implementation of FreeBSD is in fact losing to other flavors of Linux but the networking customized OS' (not even just thinking of Netflix at this point) are still superior.
  - drewg123 5y ago
    Most of our customizations and modifications have been upstreamed. We get similar performance on production 100g and 200g boxes when running the upstream kernel. I see no reason I shouldn't be able to hit 380Gb/s on a single-socket Rome box with an upstream kernel. I just haven't tried yet. Most of the changes that I have for the 700g number are changes to implement Disk centric NUMA siloing, which I would never upstream at this point because they are a pile of hacks. They are needed in order to change the NUMA node where memory is DMAed, so as to better utilize the xGMI links between AMD CPUs.
  - ksec 5y ago
    You can search FreeBSD and Netflix on HN. They have presented their finding over the years with detailed comments in those threads.