14 ms·

This is kind of an urban legend, do you think the multi millions servers from Google, Amazon etc... have those performance issues?

by Thaxll 3y ago

jeffbee 3y ago
The big guys don't have the patience to wait for Linux kernel networking to be fast and scalable. They bypass the kernel and take over the hardware. https://blog.acolyer.org/2019/11/11/snap-networking/ https://blog.acolyer.org/2019/11/11/snap-networking/
- corbet 3y ago
  That's funny ... the "big guys" are some of the biggest contributors to the Linux network stack, almost as if they were actually using it and cared about how well it works.
  - jeffbee 3y ago
    History has shown that tons of Linux networking scalability and performance contributions have been rejected by the gatekeepers/maintainers. The upstream kernel remains unsuitable for datacenter use, and all the major operators bypass or patch it.
    - sophacles 3y ago
      All the major operators sometimes bypass or patch it for some use cases. For others they use it as is. For other still they laugh at you for taking the type of drugs that makes one think any CPU is sufficient to handle networking in code. Networking isn't a one size fits all thing - different networks have different needs, and different systems in any network will have different needs. Userland networking is great until you start needing to deal with weird flows or unexpected traffic - then you end up either needing something a bit more robust and your performance starts dropping because you added a bunch of branches to your code or switched over to a kernel implementation that handles those cases. I've seen a few cases of userland networking being slower than just doing the kernel - and being kept because sometimes the what you care about is control over packet lifecycle more than raw throughput. Kernels prioritize robust network stacks that can handle a lot of cases good enough. Different implementations handle different scenarios better - there's plenty of very high performance networking done with vanilla linux and vanilla freebsd.
    - eddtests 3y ago
      Do you have links on this? I’ve not heard anything about it
      - tptacek 3y ago
        I believe they're paraphrasing the Snap paper, and also that they're extrapolating too far from it.
    - suprjami 3y ago
      Look who you're arguing with.
- tptacek 3y ago
  Over the course of several years, the architecture underpinning Snap has been used in production for multiple networking applications, including network virtualization for cloud VMs [19], packet-processing for Internet peering [62], scalable load balancing [22], and Pony Express, a reliable transport and communications stack that is our focus for the remainder of this paper. This paper suggests, as I would have expected, that Google uses userland networking in strategic spots where low-level network development is important (SDNs and routing), and not for normal applications.
  - jeffbee 3y ago
    "and Pony Express" is the operative phrase. As the paper states on page 1, "Snap is deployed to over half of our fleet of machines and supports the needs of numerous teams." According to the paper it is not niche.
    - tptacek 3y ago
      Isn't Pony Express a ground-up replacement for all of TCP/IP? It doesn't even present a TCP/UDP socket interface.
      - jeffbee 3y ago
        Correct. That is my point. The sockets interface, and design choices within the Linux kernel, make ordinary TCP sockets too difficult to exploit in a datacenter environment. The general trend is away from TCP sockets. QUIC (HTTP/3) is a less extreme retreat from TCP, moving all the flow control, congestion, and retry logic out of the kernel and into the application. An example of how Linux TCP is unsuitable for datacenters is that the minimum RTO is hard-coded to 200ms, which is essentially forever. People have been trying to land better or at least more configurable parameters upstream for decades. I am hardly the first person to point out the deficiencies. Google presented tuning Linux for datacenter applications at LPC 2022, and their deck has barely changed in 15 years.
        tptacek 3y ago
dijit 3y ago
This is "appeal to authority" fallacy incarnate. Google/Amazon "etc;" are likely happy to pay the cost because it really is "good enough" and the benefits of Linux over FreeBSD are otherwise quite considerable. Google in particular seems blissfully happy to literally throw hardware at problems; since hardware is (for them especially) fundamentally extremely cheap. Even multiple percentage gains in throughput are not necessary for most applications, and Linux is decent enough with latency if you avoid having complex IP/NFTables rules and avoid CONNTRACK like the plague. as u/jeffbee says anyway, most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking.
- Thaxll 3y ago
  I know they bypass the kernel but my point still stands, most of the servers on the internet runs on Linux, that's a fact, so there was more money, time invested, man power on that OS than any others.
  - dijit 3y ago
    Your point is that popularity means that it will improve. This is true, to a point. Counterpoint: Windows Desktop Experience. EDIT: that comment was glib, let me do a proper counterpoint. Common area's are some of the most least maintained in reality; I can think of meet-me-rooms or central fibre hubs in major cities; they are expensive and subject to a lot of the whims of the major provider. Crucially, despite large amounts of investment, the underlying architecture or infrastructure remains, even if the entire fabric of the area changes around it. Most providers using these kind of common areas do everything they can to avoid touching the area itself, especially as after a while it becomes very difficult to navigate and politically charged. Fundamentally the architecture of Linux's network stack, really is, "good enough", which is almost worse than you would originally think since "good enough" means there's no reason to look there. There is an old parable about "worse is better" because if something is truly broken people will put effort into fixing it. Linux's networking stack is fine, it's just not quite as good an architecture as the FreeBSD one. FreeBSD one has a lot less attention on it but fundamentally it's a cleaner implementation and easier to get much more out of.. You will find the same argument ad infinitum regarding other subjects such as Epoll vs IOCP vs kqueue (Epoll was abysmally terrible though and ended up being replaced by IO_URING, but even that took over a decade)
ori_b 3y ago
Do you think that (outside of a few special cases) they're using anything near the network bandwidth available to them? I would expect in the 1% to 10% bandwidth utilization, on average. From my vague recollection, that's what it was at FB when I was there. They put stupid amounts of network capacity in so that the engineers rarely have to think about the capacity of the links they're using, and that if their needs grow, they're not bottlenecked on a build out. To answer the original question, it's complicated. I have a weird client where freebsd gets 450 MiB/s, and Linux gets 85 with the default congestion control algorithm. Changing the congestion control algorithm can get me between 1.7 MiB/s and 470 MiB/s. So, better performance... Under what circumstances?
toast0 3y ago
I've been at two large FreeBSD users that switched to Linux, and the reason was never performance. Yahoo switched to Linux because having a single server OS is nicer than having two, acquisitions were nearly all running Linux, and Linux was at least good enough (although having all of Overture crash at the same time because of a leap second bug in the Linux kernel, twice, wasn't great), and maybe something about easier to hire for kernel engineers. WhatsApp switched to Linux because the server team was mostly ex-Yahoo and had seen how much strife running a different OS than your acquirer in their datacenters causes. There was enough non-negotiable strife from all of the other tech and philosophy differences, that accepting the acquirer kernel that is at least good enough was worthwhile. I'm not saying FreeBSD or Linux has better performance. I like the reputation FreeBSD has for network performance, and it certainly has good performance, but is it better than Linux? I don't know, I never ran apples to apples comparisons, because whenever I was involved in a switch, the hardware was going to be very different, and the changeover was a policy decision rather than a technical one. Personally, I run FreeBSD when I can, and Linux when I have to. FreeBSD's development model and smaller team lead to less churn, and I value stability and consistency. Other people have different values, and that's fine too.

This is kind of an urban legend, do you think the multi millions servers from Google, Amazon etc... have those performance issues?

by Thaxll 3y ago

This is kind of an urban legend, do you think the multi millions servers from Google, Amazon etc... have those performance issues?

jeffbee 3y ago

The big guys don't have the patience to wait for Linux kernel networking to be fast and scalable. They bypass the kernel and take over the hardware. https://blog.acolyer.org/2019/11/11/snap-networking/ https://blog.acolyer.org/2019/11/11/snap-networking/

corbet 3y ago
That's funny ... the "big guys" are some of the biggest contributors to the Linux network stack, almost as if they were actually using it and cared about how well it works.
- jeffbee 3y ago
  History has shown that tons of Linux networking scalability and performance contributions have been rejected by the gatekeepers/maintainers. The upstream kernel remains unsuitable for datacenter use, and all the major operators bypass or patch it.
  - sophacles 3y ago
    All the major operators sometimes bypass or patch it for some use cases. For others they use it as is. For other still they laugh at you for taking the type of drugs that makes one think any CPU is sufficient to handle networking in code. Networking isn't a one size fits all thing - different networks have different needs, and different systems in any network will have different needs. Userland networking is great until you start needing to deal with weird flows or unexpected traffic - then you end up either needing something a bit more robust and your performance starts dropping because you added a bunch of branches to your code or switched over to a kernel implementation that handles those cases. I've seen a few cases of userland networking being slower than just doing the kernel - and being kept because sometimes the what you care about is control over packet lifecycle more than raw throughput. Kernels prioritize robust network stacks that can handle a lot of cases good enough. Different implementations handle different scenarios better - there's plenty of very high performance networking done with vanilla linux and vanilla freebsd.
  - eddtests 3y ago
    Do you have links on this? I’ve not heard anything about it
    - tptacek 3y ago
      I believe they're paraphrasing the Snap paper, and also that they're extrapolating too far from it.
  - suprjami 3y ago
    Look who you're arguing with.
tptacek 3y ago
Over the course of several years, the architecture underpinning Snap has been used in production for multiple networking applications, including network virtualization for cloud VMs [19], packet-processing for Internet peering [62], scalable load balancing [22], and Pony Express, a reliable transport and communications stack that is our focus for the remainder of this paper. This paper suggests, as I would have expected, that Google uses userland networking in strategic spots where low-level network development is important (SDNs and routing), and not for normal applications.
- jeffbee 3y ago
  "and Pony Express" is the operative phrase. As the paper states on page 1, "Snap is deployed to over half of our fleet of machines and supports the needs of numerous teams." According to the paper it is not niche.
  - tptacek 3y ago
    Isn't Pony Express a ground-up replacement for all of TCP/IP? It doesn't even present a TCP/UDP socket interface.
    - jeffbee 3y ago
      Correct. That is my point. The sockets interface, and design choices within the Linux kernel, make ordinary TCP sockets too difficult to exploit in a datacenter environment. The general trend is away from TCP sockets. QUIC (HTTP/3) is a less extreme retreat from TCP, moving all the flow control, congestion, and retry logic out of the kernel and into the application. An example of how Linux TCP is unsuitable for datacenters is that the minimum RTO is hard-coded to 200ms, which is essentially forever. People have been trying to land better or at least more configurable parameters upstream for decades. I am hardly the first person to point out the deficiencies. Google presented tuning Linux for datacenter applications at LPC 2022, and their deck has barely changed in 15 years.
      - tptacek 3y ago

dijit 3y ago

This is "appeal to authority" fallacy incarnate. Google/Amazon "etc;" are likely happy to pay the cost because it really is "good enough" and the benefits of Linux over FreeBSD are otherwise quite considerable. Google in particular seems blissfully happy to literally throw hardware at problems; since hardware is (for them especially) fundamentally extremely cheap. Even multiple percentage gains in throughput are not necessary for most applications, and Linux is decent enough with latency if you avoid having complex IP/NFTables rules and avoid CONNTRACK like the plague. as u/jeffbee says anyway, most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking.

Thaxll 3y ago
I know they bypass the kernel but my point still stands, most of the servers on the internet runs on Linux, that's a fact, so there was more money, time invested, man power on that OS than any others.
- dijit 3y ago
  Your point is that popularity means that it will improve. This is true, to a point. Counterpoint: Windows Desktop Experience. EDIT: that comment was glib, let me do a proper counterpoint. Common area's are some of the most least maintained in reality; I can think of meet-me-rooms or central fibre hubs in major cities; they are expensive and subject to a lot of the whims of the major provider. Crucially, despite large amounts of investment, the underlying architecture or infrastructure remains, even if the entire fabric of the area changes around it. Most providers using these kind of common areas do everything they can to avoid touching the area itself, especially as after a while it becomes very difficult to navigate and politically charged. Fundamentally the architecture of Linux's network stack, really is, "good enough", which is almost worse than you would originally think since "good enough" means there's no reason to look there. There is an old parable about "worse is better" because if something is truly broken people will put effort into fixing it. Linux's networking stack is fine, it's just not quite as good an architecture as the FreeBSD one. FreeBSD one has a lot less attention on it but fundamentally it's a cleaner implementation and easier to get much more out of.. You will find the same argument ad infinitum regarding other subjects such as Epoll vs IOCP vs kqueue (Epoll was abysmally terrible though and ended up being replaced by IO_URING, but even that took over a decade)

ori_b 3y ago

Do you think that (outside of a few special cases) they're using anything near the network bandwidth available to them? I would expect in the 1% to 10% bandwidth utilization, on average. From my vague recollection, that's what it was at FB when I was there. They put stupid amounts of network capacity in so that the engineers rarely have to think about the capacity of the links they're using, and that if their needs grow, they're not bottlenecked on a build out. To answer the original question, it's complicated. I have a weird client where freebsd gets 450 MiB/s, and Linux gets 85 with the default congestion control algorithm. Changing the congestion control algorithm can get me between 1.7 MiB/s and 470 MiB/s. So, better performance... Under what circumstances?

toast0 3y ago

I've been at two large FreeBSD users that switched to Linux, and the reason was never performance. Yahoo switched to Linux because having a single server OS is nicer than having two, acquisitions were nearly all running Linux, and Linux was at least good enough (although having all of Overture crash at the same time because of a leap second bug in the Linux kernel, twice, wasn't great), and maybe something about easier to hire for kernel engineers. WhatsApp switched to Linux because the server team was mostly ex-Yahoo and had seen how much strife running a different OS than your acquirer in their datacenters causes. There was enough non-negotiable strife from all of the other tech and philosophy differences, that accepting the acquirer kernel that is at least good enough was worthwhile. I'm not saying FreeBSD or Linux has better performance. I like the reputation FreeBSD has for network performance, and it certainly has good performance, but is it better than Linux? I don't know, I never ran apples to apples comparisons, because whenever I was involved in a switch, the hardware was going to be very different, and the changeover was a policy decision rather than a technical one. Personally, I run FreeBSD when I can, and Linux when I have to. FreeBSD's development model and smaller team lead to less churn, and I value stability and consistency. Other people have different values, and that's fine too.