9 ms·

Has Linux reached parity with BSD in terms of the TCP stack? My understanding was that it still wasn't as efficient but that info is outdated.

brendangregg 10y ago
I'd have to recheck. Linux has done some interesting new systems engineering work with BPF kernel bypass to improve network performance (the eXtreme Data Path project, used by Facebook).
dekhn 10y ago
"""Has Linux reached parity with BSD in terms of the TCP stack?""" Im not sure what you mean. Linux has led TCP implementations for a decade now.
- ryao 10y ago
  Two years ago, Facebook was trying to hire someone to make Linux's network stack as good as FreeBSD's: http://m.slashdot.org/story/205565 http://m.slashdot.org/story/205565
  - brendangregg 10y ago
    That's a great find. And look at what Facebook have done: invested in BPF and eXtreme Data Path.
    - dekhn 10y ago
      that job posting provides no specific details about how BSD is "better" than Linux execpt making unspecific allegations about stability.
      - ryao 10y ago
        I imagine that it worked better in their testing and rather than jump ship, they decided to invest in making Linux better. If they knew how FreeBSD was better at that time, they likely would have patched Linux rather than look for talent that could figure it out.
  - dekhn 10y ago
    I work for Google and previously worked at LBL on a team that developed optimized network stacks. I have a fair amount of experience running large scale data transfers as well as low-latency network in LANs. The Linux network stack is great. It's the preferred system of choice for nearly every researcher in the networking field. I don't know what Facebook meant in their case.
    - ryao 10y ago
      You have more experience than I do in this area, but I am going to reply anyway in the belief that I can contribute some useful information to the discussion. First, there is a discussion of this here: https://www.quora.com/How-is-FreeBSDs-network-stack-superior-to-that-of-Linux https://www.quora.com/How-is-FreeBSDs-network-stack-superior... The main remark seems to be: > The predominant difference is that the FreeBSD network stack was much more carefully designed. The Linux stack was less careful and thus is much more haphazard. Also, more work has been put into optimizing the FreeBSD stack. It is not my area of expertise, but the Linux skbuf seems to fit the description of haphazard while the FreeBSD mbuf seems to fit the description of more carefully designed. The same could be said about epoll versus kqueue. The remark about more work in optimizing the FreeBSD stack also seems to be true. While I cannot speak for everything in FreeBSD's network stack, I do know that FreeBSD's netmap far exceeded anything Linux could do at the time and while it is available on Linux, I never hear of it being used anywhere but on FreeBSD: http://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/ Development of FreeBSD's network stack had plenty of innovative things in development at the time Facebook's post was made: https://wiki.freebsd.org/201405DevSummit/NetworkStack https://wiki.freebsd.org/201405DevSummit/NetworkStack That included additional contributions from a major network equipment vendor that had made many contributions throughout the years. If I checked the commit history, I imagine I would find performance work done by said vendor. From what I can tell, FreeBSD's network stack is improving regardless of whether the rest of us hear about it. Lastly, there have been multiple things discovered to be wrong in the Linux network stack since that facebook job listing. Two prominent ones that I recall offhand are: https://blog.cloudflare.com/the-story-of-one-latency-spike/ https://blog.cloudflare.com/the-story-of-one-latency-spike/ https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-team-for-open-source.html https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-tea... They both could fall into the category of stability problems to which facebook had alluded. The second one more so though: > The end result is that applications that oscillate between transmitting lots of data and then laying quiescent for a bit before returning to high rates of sending will transmit way too fast when returning to the sending state. This consequence of this is self induced packet loss along with retransmissions, wasted bandwidth, out of order packet delivery, and application level stalls.
walrus01 10y ago
Maybe ten years ago when the first generation of PCI-X Intel chipset 10GbE NICs were available for servers, the driver support and tcp offload engine support were much better on FreeBSD. But the situation is reversed now with all of the drivers and updates to the v4.x series Linux kernel.
SEJeff 10y ago
Linux has been beating BSD for at least 8-10 years when it comes to TCP. When it comes to new features in TCP-land, Linux easily beats it. Google added Receive Side Scaling / Receive Flow Steering to Linux years ago, and it is still a WIP in FreeBSD as an example. Also take a look at much of the bufferbloat research recently that has been merged into Linux, etc.
- matt_wulfeck 10y ago
  I guess my information was not just outdated, but clearly wrong.
  - SEJeff 10y ago
    Don't get me wrong, BSD is still absolutely solid, but for anything cutting edge, Linux is spanking the pants off of it.
    - kev009 10y ago
      This is an adolescent evaluation. FreeBSD will have a new TCP stack with BBR made public in a couple months. It will be easier to correctly deploy and more cohesive than Linux. The entire packet path is more cohesive and easier to debug and tune using DTrace although Linux might have caught up here recently. By volume, FreeBSD is doing at least 30% of Internet facing traffic between a well known company and some quieter giants. BTW it only took a half dozen people collaborating between 3 companies about 2 years catch up to state of the art. I've done a great deal of reading and research on OS ethos, IMO a thriving and production worthy operating system can be maintained with as few as 40 people in total. The superiority of Linux feels exaggerated, and systems innovation has chilled because of it.
    - emaste 10y ago
      As is often the case it depends on the specifics of the application and on those building a solution. As far as raw performance is concerned FreeBSD performs very well. Netflix gets nearly 100Gbps from storage out the network on their FreeBSD+NGINX OCA appliances. Some details in the "Mellanox CDN Reference Architecture" whitepaper at http://www.mellanox.com/related-docs/solutions/cdn_ref_arch.pdf http://www.mellanox.com/related-docs/solutions/cdn_ref_arch..... The closest equivalent I've found on Linux was a blog post on BBC streaming getting about 1/4 of the performance. Chelsio has a demo video (with terrible music) using TCP zero copy of 100Gbps on a single TCP session, with <1% CPU usage https://www.youtube.com/watch?v=NKTApBf8Oko https://www.youtube.com/watch?v=NKTApBf8Oko. At SC16 NASA had a "Building Cost-Effective 100-Gbps Firewalls for HPC" demo, using FreeBSD and netmap: https://www.nas.nasa.gov/SC16/demos/demo9.html https://www.nas.nasa.gov/SC16/demos/demo9.html

brendangregg 10y ago

I'd have to recheck. Linux has done some interesting new systems engineering work with BPF kernel bypass to improve network performance (the eXtreme Data Path project, used by Facebook).

dekhn 10y ago

"""Has Linux reached parity with BSD in terms of the TCP stack?""" Im not sure what you mean. Linux has led TCP implementations for a decade now.

ryao 10y ago
Two years ago, Facebook was trying to hire someone to make Linux's network stack as good as FreeBSD's: http://m.slashdot.org/story/205565 http://m.slashdot.org/story/205565
- brendangregg 10y ago
  That's a great find. And look at what Facebook have done: invested in BPF and eXtreme Data Path.
  - dekhn 10y ago
    that job posting provides no specific details about how BSD is "better" than Linux execpt making unspecific allegations about stability.
    - ryao 10y ago
      I imagine that it worked better in their testing and rather than jump ship, they decided to invest in making Linux better. If they knew how FreeBSD was better at that time, they likely would have patched Linux rather than look for talent that could figure it out.
- dekhn 10y ago
  I work for Google and previously worked at LBL on a team that developed optimized network stacks. I have a fair amount of experience running large scale data transfers as well as low-latency network in LANs. The Linux network stack is great. It's the preferred system of choice for nearly every researcher in the networking field. I don't know what Facebook meant in their case.
  - ryao 10y ago
    You have more experience than I do in this area, but I am going to reply anyway in the belief that I can contribute some useful information to the discussion. First, there is a discussion of this here: https://www.quora.com/How-is-FreeBSDs-network-stack-superior-to-that-of-Linux https://www.quora.com/How-is-FreeBSDs-network-stack-superior... The main remark seems to be: > The predominant difference is that the FreeBSD network stack was much more carefully designed. The Linux stack was less careful and thus is much more haphazard. Also, more work has been put into optimizing the FreeBSD stack. It is not my area of expertise, but the Linux skbuf seems to fit the description of haphazard while the FreeBSD mbuf seems to fit the description of more carefully designed. The same could be said about epoll versus kqueue. The remark about more work in optimizing the FreeBSD stack also seems to be true. While I cannot speak for everything in FreeBSD's network stack, I do know that FreeBSD's netmap far exceeded anything Linux could do at the time and while it is available on Linux, I never hear of it being used anywhere but on FreeBSD: http://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/ Development of FreeBSD's network stack had plenty of innovative things in development at the time Facebook's post was made: https://wiki.freebsd.org/201405DevSummit/NetworkStack https://wiki.freebsd.org/201405DevSummit/NetworkStack That included additional contributions from a major network equipment vendor that had made many contributions throughout the years. If I checked the commit history, I imagine I would find performance work done by said vendor. From what I can tell, FreeBSD's network stack is improving regardless of whether the rest of us hear about it. Lastly, there have been multiple things discovered to be wrong in the Linux network stack since that facebook job listing. Two prominent ones that I recall offhand are: https://blog.cloudflare.com/the-story-of-one-latency-spike/ https://blog.cloudflare.com/the-story-of-one-latency-spike/ https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-team-for-open-source.html https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-tea... They both could fall into the category of stability problems to which facebook had alluded. The second one more so though: > The end result is that applications that oscillate between transmitting lots of data and then laying quiescent for a bit before returning to high rates of sending will transmit way too fast when returning to the sending state. This consequence of this is self induced packet loss along with retransmissions, wasted bandwidth, out of order packet delivery, and application level stalls.

walrus01 10y ago

Maybe ten years ago when the first generation of PCI-X Intel chipset 10GbE NICs were available for servers, the driver support and tcp offload engine support were much better on FreeBSD. But the situation is reversed now with all of the drivers and updates to the v4.x series Linux kernel.

SEJeff 10y ago

Linux has been beating BSD for at least 8-10 years when it comes to TCP. When it comes to new features in TCP-land, Linux easily beats it. Google added Receive Side Scaling / Receive Flow Steering to Linux years ago, and it is still a WIP in FreeBSD as an example. Also take a look at much of the bufferbloat research recently that has been merged into Linux, etc.

matt_wulfeck 10y ago
I guess my information was not just outdated, but clearly wrong.
- SEJeff 10y ago
  Don't get me wrong, BSD is still absolutely solid, but for anything cutting edge, Linux is spanking the pants off of it.
  - kev009 10y ago
    This is an adolescent evaluation. FreeBSD will have a new TCP stack with BBR made public in a couple months. It will be easier to correctly deploy and more cohesive than Linux. The entire packet path is more cohesive and easier to debug and tune using DTrace although Linux might have caught up here recently. By volume, FreeBSD is doing at least 30% of Internet facing traffic between a well known company and some quieter giants. BTW it only took a half dozen people collaborating between 3 companies about 2 years catch up to state of the art. I've done a great deal of reading and research on OS ethos, IMO a thriving and production worthy operating system can be maintained with as few as 40 people in total. The superiority of Linux feels exaggerated, and systems innovation has chilled because of it.
  - emaste 10y ago
    As is often the case it depends on the specifics of the application and on those building a solution. As far as raw performance is concerned FreeBSD performs very well. Netflix gets nearly 100Gbps from storage out the network on their FreeBSD+NGINX OCA appliances. Some details in the "Mellanox CDN Reference Architecture" whitepaper at http://www.mellanox.com/related-docs/solutions/cdn_ref_arch.pdf http://www.mellanox.com/related-docs/solutions/cdn_ref_arch..... The closest equivalent I've found on Linux was a blog post on BBC streaming getting about 1/4 of the performance. Chelsio has a demo video (with terrible music) using TCP zero copy of 100Gbps on a single TCP session, with <1% CPU usage https://www.youtube.com/watch?v=NKTApBf8Oko https://www.youtube.com/watch?v=NKTApBf8Oko. At SC16 NASA had a "Building Cost-Effective 100-Gbps Firewalls for HPC" demo, using FreeBSD and netmap: https://www.nas.nasa.gov/SC16/demos/demo9.html https://www.nas.nasa.gov/SC16/demos/demo9.html