3 ms·

Linux has been beating BSD for at least 8-10 years when it comes to TCP. When it comes to new features in TCP-land, Linux easily beats it. Google added Receive

by SEJeff 10y ago

Linux has been beating BSD for at least 8-10 years when it comes to TCP. When it comes to new features in TCP-land, Linux easily beats it. Google added Receive Side Scaling / Receive Flow Steering to Linux years ago, and it is still a WIP in FreeBSD as an example. Also take a look at much of the bufferbloat research recently that has been merged into Linux, etc.

matt_wulfeck 10y ago
I guess my information was not just outdated, but clearly wrong.
- SEJeff 10y ago
  Don't get me wrong, BSD is still absolutely solid, but for anything cutting edge, Linux is spanking the pants off of it.
  - kev009 10y ago
    This is an adolescent evaluation. FreeBSD will have a new TCP stack with BBR made public in a couple months. It will be easier to correctly deploy and more cohesive than Linux. The entire packet path is more cohesive and easier to debug and tune using DTrace although Linux might have caught up here recently. By volume, FreeBSD is doing at least 30% of Internet facing traffic between a well known company and some quieter giants. BTW it only took a half dozen people collaborating between 3 companies about 2 years catch up to state of the art. I've done a great deal of reading and research on OS ethos, IMO a thriving and production worthy operating system can be maintained with as few as 40 people in total. The superiority of Linux feels exaggerated, and systems innovation has chilled because of it.
  - emaste 10y ago
    As is often the case it depends on the specifics of the application and on those building a solution. As far as raw performance is concerned FreeBSD performs very well. Netflix gets nearly 100Gbps from storage out the network on their FreeBSD+NGINX OCA appliances. Some details in the "Mellanox CDN Reference Architecture" whitepaper at http://www.mellanox.com/related-docs/solutions/cdn_ref_arch.pdf http://www.mellanox.com/related-docs/solutions/cdn_ref_arch..... The closest equivalent I've found on Linux was a blog post on BBC streaming getting about 1/4 of the performance. Chelsio has a demo video (with terrible music) using TCP zero copy of 100Gbps on a single TCP session, with <1% CPU usage https://www.youtube.com/watch?v=NKTApBf8Oko https://www.youtube.com/watch?v=NKTApBf8Oko. At SC16 NASA had a "Building Cost-Effective 100-Gbps Firewalls for HPC" demo, using FreeBSD and netmap: https://www.nas.nasa.gov/SC16/demos/demo9.html https://www.nas.nasa.gov/SC16/demos/demo9.html
    - SEJeff 10y ago
      You're spot on regarding the app and FreeBSD performing very well. Don't disagree with you one bit. Also, great link on the Netflix CDN work, they're doing some really fascinating stuff. It is nice to see the openness. I work directly with both of the gents who gave this talk about 100G networking[1] (on Linux) and still find that much of the actual cutting edge research is done on Linux. Perhaps I'm biased! I've also been to one of Mellanox's engineering offices (Tel Aviv) to speak with their engineers at my previous employer 7-8 years ago. They told me they do most all of their prototyping and initial development on Linux, and RHEL to be specific. Then then port to other platforms. Maybe I was wrong on some of this, but my use case (due to my employer's industry being finance) is lower latency, where Linux absolutely and positively crushes anything else. [1] http://events.linuxfoundation.org/sites/events/files/slides/100G%20Networking%20Toronto_0.pdf
      - kev009 10y ago
        Mellanox is now one of the role model vendors in the FreeBSD ecosystem. They have a handful of BSD developers as well as sales and support staff that are in tune with the needs of high scalability FreeBSD users.
      - Annatar 10y ago
        Maybe I was wrong on some of this, but my use case (due to my employer's industry being finance) is lower latency, where Linux absolutely and positively crushes anything else. Actually, while we're on the subject, SmartOS with CPU bursting from illumos is the leader in low latency trading: http://containersummit.io/events/sf-2015/videos/wolf-of-what-containers-on-wall-street http://containersummit.io/events/sf-2015/videos/wolf-of-what...
        SEJeff 10y ago
        That is a slick platform they've built, but I still don't see how it is competitive with Linux for very low latencies. He mentions trading at microseconds, but we're building microwave radio networks to trade at nanoseconds. Unless this has changed extremely recently, Solaris/Illumos and hence SmartOS still don't have tickless kernels. I recall Solaris having a 100hz tick by default which you could change to 1000hz with a special boot flag. Linux has had dynticks since fairly early 2.6 kernels and with the modern 3.x kernels (RHEL7+), you've got full on tickless via the nohz_full options. Without this, the kernel interrupts the application to use cpu time. Additionally, I don't believe (Experts please correct me if this is wrong) SmartOS has an equivalent to Linux's isolcpus boot command line flag (or cpu_exclusive=1 if you're in a cpuset) to remove a cpu core entirely from the global scheduler domain. This prevents any tasks from running on that CPU, including kernel threads. Kernel threads will still occasionally interrupt applications if you simply set the affinity on pid 1 so that does't count. These two features, along with hardware that is configured to not throw SMIs, allow Linux to get out of the way of applications for truly low latency. As far as I'm aware, this is impossible to do in Solaris/SmartOS. I'm not even getting into the SLUB memory allocator being better or the lazy TLB in Linux massively lowering TLB shootdowns, etc, etc. There is a reason why virtually every single major financial exchange in the world runs Linux (CME in Chicago, NYSE/NYMEX in New York, LSE in London, and Xetra in Frankfurt), it is better for the low latency use case.
kev009 10y ago
What's WIP about FreeBSD's RSS?
drewg123 10y ago
The RSS is Linux is not particularly useful (for netflix scale workloads) because it does not integrate the RSS hashing across the entire stack. So all you get is connection sharding. With real RSS, as done in Windows and FreeBSD where the kernel has intimate knowledge of the hash key and algorithm, you can use RSS to split the TCP hash table up and make it per-CPU. By using multiple accept sockets for per-CPU workers, you can effectively keep everything for a single connection on a CPU, and run almost everything with no cross-CPU contention. You can't move the connection around at will between CPUs, but you don't care to, because no connection is special (in a netflix workload), it just one of tens of thousands. Adrian Chadd did most of the FreeBSD RSS work, and gave a good talk about it at BAFUG: https://www.youtube.com/watch?v=7CvIztTz-RQ https://www.youtube.com/watch?v=7CvIztTz-RQ The RSS in Linux was just used for load spreading (the last I checked, I haven't used Linux much since I left Google 1.5 years ago). If this has improved, I'd love to hear about it. Linux RFS depends on the packets being dispatched to the correct CPU for the connection by the interrupt handler running wherever the packet happened to land. This has cache & memory locality implications, especially on NUMA. Linux aRFS lets the NIC do the steering. Unfortunately, each connection requires an interaction with the NIC to poke it into the steering table, and most NICs can't steer 100,000 connections. So, to sum up, Linux has a lot of cool tech for steering individual connections and support for that varies greatly by NIC. Windows and FreeBSD use standard RSS to predictably steer an unlimited number of connections. For a large CDN server, the latter is more useful. However, for low-latency / high bandwidth applications, I can see the advantage to aRFS.
ryao 10y ago
I wouldn't say 8 to 10 years. A major bug in Linux's default congestion control algorithm was only fixed just last year: https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-team-for-open-source.html https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-tea... Linux is the platform of choice for bufferbloat research, although FreeBSD isn't far behind in adopting the results of it: https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-April/006154.html https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-April/...

by SEJeff 10y ago

matt_wulfeck 10y ago

I guess my information was not just outdated, but clearly wrong.

SEJeff 10y ago
Don't get me wrong, BSD is still absolutely solid, but for anything cutting edge, Linux is spanking the pants off of it.
- kev009 10y ago
  This is an adolescent evaluation. FreeBSD will have a new TCP stack with BBR made public in a couple months. It will be easier to correctly deploy and more cohesive than Linux. The entire packet path is more cohesive and easier to debug and tune using DTrace although Linux might have caught up here recently. By volume, FreeBSD is doing at least 30% of Internet facing traffic between a well known company and some quieter giants. BTW it only took a half dozen people collaborating between 3 companies about 2 years catch up to state of the art. I've done a great deal of reading and research on OS ethos, IMO a thriving and production worthy operating system can be maintained with as few as 40 people in total. The superiority of Linux feels exaggerated, and systems innovation has chilled because of it.
- emaste 10y ago
  As is often the case it depends on the specifics of the application and on those building a solution. As far as raw performance is concerned FreeBSD performs very well. Netflix gets nearly 100Gbps from storage out the network on their FreeBSD+NGINX OCA appliances. Some details in the "Mellanox CDN Reference Architecture" whitepaper at http://www.mellanox.com/related-docs/solutions/cdn_ref_arch.pdf http://www.mellanox.com/related-docs/solutions/cdn_ref_arch..... The closest equivalent I've found on Linux was a blog post on BBC streaming getting about 1/4 of the performance. Chelsio has a demo video (with terrible music) using TCP zero copy of 100Gbps on a single TCP session, with <1% CPU usage https://www.youtube.com/watch?v=NKTApBf8Oko https://www.youtube.com/watch?v=NKTApBf8Oko. At SC16 NASA had a "Building Cost-Effective 100-Gbps Firewalls for HPC" demo, using FreeBSD and netmap: https://www.nas.nasa.gov/SC16/demos/demo9.html https://www.nas.nasa.gov/SC16/demos/demo9.html
  - SEJeff 10y ago
    You're spot on regarding the app and FreeBSD performing very well. Don't disagree with you one bit. Also, great link on the Netflix CDN work, they're doing some really fascinating stuff. It is nice to see the openness. I work directly with both of the gents who gave this talk about 100G networking[1] (on Linux) and still find that much of the actual cutting edge research is done on Linux. Perhaps I'm biased! I've also been to one of Mellanox's engineering offices (Tel Aviv) to speak with their engineers at my previous employer 7-8 years ago. They told me they do most all of their prototyping and initial development on Linux, and RHEL to be specific. Then then port to other platforms. Maybe I was wrong on some of this, but my use case (due to my employer's industry being finance) is lower latency, where Linux absolutely and positively crushes anything else. [1] http://events.linuxfoundation.org/sites/events/files/slides/100G%20Networking%20Toronto_0.pdf
    - kev009 10y ago
      Mellanox is now one of the role model vendors in the FreeBSD ecosystem. They have a handful of BSD developers as well as sales and support staff that are in tune with the needs of high scalability FreeBSD users.
    - Annatar 10y ago
      Maybe I was wrong on some of this, but my use case (due to my employer's industry being finance) is lower latency, where Linux absolutely and positively crushes anything else. Actually, while we're on the subject, SmartOS with CPU bursting from illumos is the leader in low latency trading: http://containersummit.io/events/sf-2015/videos/wolf-of-what-containers-on-wall-street http://containersummit.io/events/sf-2015/videos/wolf-of-what...
      - SEJeff 10y ago
        That is a slick platform they've built, but I still don't see how it is competitive with Linux for very low latencies. He mentions trading at microseconds, but we're building microwave radio networks to trade at nanoseconds. Unless this has changed extremely recently, Solaris/Illumos and hence SmartOS still don't have tickless kernels. I recall Solaris having a 100hz tick by default which you could change to 1000hz with a special boot flag. Linux has had dynticks since fairly early 2.6 kernels and with the modern 3.x kernels (RHEL7+), you've got full on tickless via the nohz_full options. Without this, the kernel interrupts the application to use cpu time. Additionally, I don't believe (Experts please correct me if this is wrong) SmartOS has an equivalent to Linux's isolcpus boot command line flag (or cpu_exclusive=1 if you're in a cpuset) to remove a cpu core entirely from the global scheduler domain. This prevents any tasks from running on that CPU, including kernel threads. Kernel threads will still occasionally interrupt applications if you simply set the affinity on pid 1 so that does't count. These two features, along with hardware that is configured to not throw SMIs, allow Linux to get out of the way of applications for truly low latency. As far as I'm aware, this is impossible to do in Solaris/SmartOS. I'm not even getting into the SLUB memory allocator being better or the lazy TLB in Linux massively lowering TLB shootdowns, etc, etc. There is a reason why virtually every single major financial exchange in the world runs Linux (CME in Chicago, NYSE/NYMEX in New York, LSE in London, and Xetra in Frankfurt), it is better for the low latency use case.

kev009 10y ago

What's WIP about FreeBSD's RSS?

drewg123 10y ago

The RSS is Linux is not particularly useful (for netflix scale workloads) because it does not integrate the RSS hashing across the entire stack. So all you get is connection sharding. With real RSS, as done in Windows and FreeBSD where the kernel has intimate knowledge of the hash key and algorithm, you can use RSS to split the TCP hash table up and make it per-CPU. By using multiple accept sockets for per-CPU workers, you can effectively keep everything for a single connection on a CPU, and run almost everything with no cross-CPU contention. You can't move the connection around at will between CPUs, but you don't care to, because no connection is special (in a netflix workload), it just one of tens of thousands. Adrian Chadd did most of the FreeBSD RSS work, and gave a good talk about it at BAFUG: https://www.youtube.com/watch?v=7CvIztTz-RQ https://www.youtube.com/watch?v=7CvIztTz-RQ The RSS in Linux was just used for load spreading (the last I checked, I haven't used Linux much since I left Google 1.5 years ago). If this has improved, I'd love to hear about it. Linux RFS depends on the packets being dispatched to the correct CPU for the connection by the interrupt handler running wherever the packet happened to land. This has cache & memory locality implications, especially on NUMA. Linux aRFS lets the NIC do the steering. Unfortunately, each connection requires an interaction with the NIC to poke it into the steering table, and most NICs can't steer 100,000 connections. So, to sum up, Linux has a lot of cool tech for steering individual connections and support for that varies greatly by NIC. Windows and FreeBSD use standard RSS to predictably steer an unlimited number of connections. For a large CDN server, the latter is more useful. However, for low-latency / high bandwidth applications, I can see the advantage to aRFS.

ryao 10y ago

I wouldn't say 8 to 10 years. A major bug in Linux's default congestion control algorithm was only fixed just last year: https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-team-for-open-source.html https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-tea... Linux is the platform of choice for bufferbloat research, although FreeBSD isn't far behind in adopting the results of it: https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-April/006154.html https://lists.freebsd.org/pipermail/freebsd-ipfw/2016-April/...