4 ms·

This is "appeal to authority" fallacy incarnate. Google/Amazon "etc;" are likely happy to pay the cost because it really is "good enough" and the benefits of L

by dijit 3y ago

This is "appeal to authority" fallacy incarnate. Google/Amazon "etc;" are likely happy to pay the cost because it really is "good enough" and the benefits of Linux over FreeBSD are otherwise quite considerable. Google in particular seems blissfully happy to literally throw hardware at problems; since hardware is (for them especially) fundamentally extremely cheap. Even multiple percentage gains in throughput are not necessary for most applications, and Linux is decent enough with latency if you avoid having complex IP/NFTables rules and avoid CONNTRACK like the plague. as u/jeffbee says anyway, most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking.

Thaxll 3y ago
I know they bypass the kernel but my point still stands, most of the servers on the internet runs on Linux, that's a fact, so there was more money, time invested, man power on that OS than any others.
- dijit 3y ago
  Your point is that popularity means that it will improve. This is true, to a point. Counterpoint: Windows Desktop Experience. EDIT: that comment was glib, let me do a proper counterpoint. Common area's are some of the most least maintained in reality; I can think of meet-me-rooms or central fibre hubs in major cities; they are expensive and subject to a lot of the whims of the major provider. Crucially, despite large amounts of investment, the underlying architecture or infrastructure remains, even if the entire fabric of the area changes around it. Most providers using these kind of common areas do everything they can to avoid touching the area itself, especially as after a while it becomes very difficult to navigate and politically charged. Fundamentally the architecture of Linux's network stack, really is, "good enough", which is almost worse than you would originally think since "good enough" means there's no reason to look there. There is an old parable about "worse is better" because if something is truly broken people will put effort into fixing it. Linux's networking stack is fine, it's just not quite as good an architecture as the FreeBSD one. FreeBSD one has a lot less attention on it but fundamentally it's a cleaner implementation and easier to get much more out of.. You will find the same argument ad infinitum regarding other subjects such as Epoll vs IOCP vs kqueue (Epoll was abysmally terrible though and ended up being replaced by IO_URING, but even that took over a decade)
  - deleted 3y ago
    [deleted]
  - Thaxll 3y ago
    Yes things improve where we're talking about multi billions $ of infra cost. Linux is not your "random on the side" feature that is good enough.
    - dijit 3y ago
      To start with: it's not that much infra cost. Especially since you don't even know what you're attempting to optimise for. Latency? p99 of linux is fine, nobody is going to care that the request took 300μs longer. Even in aggregate across a huge fleet of machines waiting an extra 3ms is totally, totally fine. Throughput? you'll bottleneck on something else most likely anyway, getting a storage array to hydrate at line rate for 100GBPs is difficult and anyway you want to do authentication and distribution of chunks and metadata operations anyway? right? You're forgetting that it's likely an additional cost of a couple million dollars per year in absolute hardware to solve that issue with throughput, which is, in TCO terms, a couple of developers. Engineering effort to replace the foundation of an OS? Probably an order of magnitude more. Definitely contains a significant amount more risk, and the potential risk of political backlash for upheaving some other companies workflow that is weird. Hardware isn't so expensive really. Of course, you could just bypass the kernel with much less effort and avoid all of this shit entirely.
deleted 3y ago
[deleted]
tptacek 3y ago
Do you know for a fact that Google primarily uses userland networking, or does that just seem accurate to you?
- dijit 3y ago
  I can't honestly answer that with the NDA I signed. However there is some public information on some components that has been shared in this thread which allows you to draw your own conclusion.
  - tptacek 3y ago
    Yes, the one link shared says essentially the opposite thing.
    - dijit 3y ago
      You may have read it wrong.
      - tptacek 3y ago
        Did I? How?
        dijit 3y ago
        For one by assuming the work that is done primarily for microkernels/appliances is the absolute limit of userspace networking at Google and that similar work would not go into a hypervisor (hypervisors which are universally treated as a vSwitch in almost all virtual environments the world over). And making that assumption when there are many public examples of Google doing this in other areas such as gVisor and Netstack?
        tptacek 3y ago
        If you have information about other userspace networking projects at Google, I'd love to read it, but the Snap paper repeatedly suggests that the userspace networking characteristics of the design are distinctive. Certainly, most networking at Google isn't netstack. Have you done much with netstack? It is many things, but ultra-high-performance isn't one of them.

by dijit 3y ago

Thaxll 3y ago

I know they bypass the kernel but my point still stands, most of the servers on the internet runs on Linux, that's a fact, so there was more money, time invested, man power on that OS than any others.

dijit 3y ago
Your point is that popularity means that it will improve. This is true, to a point. Counterpoint: Windows Desktop Experience. EDIT: that comment was glib, let me do a proper counterpoint. Common area's are some of the most least maintained in reality; I can think of meet-me-rooms or central fibre hubs in major cities; they are expensive and subject to a lot of the whims of the major provider. Crucially, despite large amounts of investment, the underlying architecture or infrastructure remains, even if the entire fabric of the area changes around it. Most providers using these kind of common areas do everything they can to avoid touching the area itself, especially as after a while it becomes very difficult to navigate and politically charged. Fundamentally the architecture of Linux's network stack, really is, "good enough", which is almost worse than you would originally think since "good enough" means there's no reason to look there. There is an old parable about "worse is better" because if something is truly broken people will put effort into fixing it. Linux's networking stack is fine, it's just not quite as good an architecture as the FreeBSD one. FreeBSD one has a lot less attention on it but fundamentally it's a cleaner implementation and easier to get much more out of.. You will find the same argument ad infinitum regarding other subjects such as Epoll vs IOCP vs kqueue (Epoll was abysmally terrible though and ended up being replaced by IO_URING, but even that took over a decade)
- deleted 3y ago
  [deleted]
- Thaxll 3y ago
  Yes things improve where we're talking about multi billions $ of infra cost. Linux is not your "random on the side" feature that is good enough.
  - dijit 3y ago
    To start with: it's not that much infra cost. Especially since you don't even know what you're attempting to optimise for. Latency? p99 of linux is fine, nobody is going to care that the request took 300μs longer. Even in aggregate across a huge fleet of machines waiting an extra 3ms is totally, totally fine. Throughput? you'll bottleneck on something else most likely anyway, getting a storage array to hydrate at line rate for 100GBPs is difficult and anyway you want to do authentication and distribution of chunks and metadata operations anyway? right? You're forgetting that it's likely an additional cost of a couple million dollars per year in absolute hardware to solve that issue with throughput, which is, in TCO terms, a couple of developers. Engineering effort to replace the foundation of an OS? Probably an order of magnitude more. Definitely contains a significant amount more risk, and the potential risk of political backlash for upheaving some other companies workflow that is weird. Hardware isn't so expensive really. Of course, you could just bypass the kernel with much less effort and avoid all of this shit entirely.

deleted 3y ago

[deleted]

tptacek 3y ago

Do you know for a fact that Google primarily uses userland networking, or does that just seem accurate to you?

dijit 3y ago
I can't honestly answer that with the NDA I signed. However there is some public information on some components that has been shared in this thread which allows you to draw your own conclusion.
- tptacek 3y ago
  Yes, the one link shared says essentially the opposite thing.
  - dijit 3y ago
    You may have read it wrong.
    - tptacek 3y ago
      Did I? How?
      - dijit 3y ago
        For one by assuming the work that is done primarily for microkernels/appliances is the absolute limit of userspace networking at Google and that similar work would not go into a hypervisor (hypervisors which are universally treated as a vSwitch in almost all virtual environments the world over). And making that assumption when there are many public examples of Google doing this in other areas such as gVisor and Netstack?
        tptacek 3y ago
        If you have information about other userspace networking projects at Google, I'd love to read it, but the Snap paper repeatedly suggests that the userspace networking characteristics of the design are distinctive. Certainly, most networking at Google isn't netstack. Have you done much with netstack? It is many things, but ultra-high-performance isn't one of them.