5 ms·
You have more experience than I do in this area, but I am going to reply anyway in the belief that I can contribute some useful information to the discussion. F
by ryao 10y ago
You have more experience than I do in this area, but I am going to reply anyway in the belief that I can contribute some useful information to the discussion. First, there is a discussion of this here:
https://www.quora.com/How-is-FreeBSDs-network-stack-superior-to-that-of-Linux https://www.quora.com/How-is-FreeBSDs-network-stack-superior...
The main remark seems to be:
> The predominant difference is that the FreeBSD network stack was much more carefully designed. The Linux stack was less careful and thus is much more haphazard. Also, more work has been put into optimizing the FreeBSD stack.
It is not my area of expertise, but the Linux skbuf seems to fit the description of haphazard while the FreeBSD mbuf seems to fit the description of more carefully designed. The same could be said about epoll versus kqueue.
The remark about more work in optimizing the FreeBSD stack also seems to be true. While I cannot speak for everything in FreeBSD's network stack, I do know that FreeBSD's netmap far exceeded anything Linux could do at the time and while it is available on Linux, I never hear of it being used anywhere but on FreeBSD:
http://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/
Development of FreeBSD's network stack had plenty of innovative things in development at the time Facebook's post was made:
https://wiki.freebsd.org/201405DevSummit/NetworkStack https://wiki.freebsd.org/201405DevSummit/NetworkStack
That included additional contributions from a major network equipment vendor that had made many contributions throughout the years. If I checked the commit history, I imagine I would find performance work done by said vendor. From what I can tell, FreeBSD's network stack is improving regardless of whether the rest of us hear about it.
Lastly, there have been multiple things discovered to be wrong in the Linux network stack since that facebook job listing. Two prominent ones that I recall offhand are:
https://blog.cloudflare.com/the-story-of-one-latency-spike/ https://blog.cloudflare.com/the-story-of-one-latency-spike/
https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-team-for-open-source.html https://bitsup.blogspot.com.au/2015/09/thanks-google-tcp-tea...
They both could fall into the category of stability problems to which facebook had alluded. The second one more so though:
> The end result is that applications that oscillate between transmitting lots of data and then laying quiescent for a bit before returning to high rates of sending will transmit way too fast when returning to the sending state. This consequence of this is self induced packet loss along with retransmissions, wasted bandwidth, out of order packet delivery, and application level stalls.
- dekhn 10y agoFor the cloudfare example they didn't actually identify any problems with Linux. What they learned- correctly- is that TCP autotuning needs to be enabled if you want high performance out of your network stack. This is covered by my previous team's page: https://fasterdata.es.net/host-tuning/linux/ https://fasterdata.es.net/host-tuning/linux/ Note: "On newer Linux OSes this is no longer needed." (IE, it's already set properly). For the second one, they fixed a bug in Linux TCP cubic implementation. FreeBSD didn't get cubic until 8.2, which was around 2009. So, you're criticizing Linux for having a in a bug in a feature that FreeBSD didn't even add until 7 years ago. Again, I will repeat: I worked on a team that did multi-OS TCP/IP optimization. What you're describing in terms of oscillation is a well-known problem in many implementations. All of the people doing research on this are now using linux as their platform for research and development.
- ryao 10y agoIf you "worked on a team that did multi-OS TCP/IP optimization", then you should know that there is no one size fits all solution that is quantitatively better for everyone. Not implementing cubic in FreeBSD when there was a bug in the only implementation of it in the world could have been an advantage in certain situations, including Facebook's. There seems to be a hubris by many Linux users that Linux is the best solution in the world for everything and it is not. There is always someone who does something better. Maybe not in everything, but the same applies to Linux. No matter how good it becomes, it is not the best in everything. Networking is a broad topic. I don't think Linux is the best in every area of networking. I am not even sure if it is the best in many of them, given that many platforms do things very well and at some point, it is hard to be better.
- X86BSD 10y agoMy opinion on this as it always has been is Linux gets some new features first before freebsd but they are always done sloppily, half assed, and start getting refined and fixed about the time freebsd implements the same feature after thinking about it, designing it well, then implementing it.
- 10y ago