7 ms·

> contain other "tweaks that are needed to run binaries that we use at Google". Most of the world manages to run their binaries on a mainline kernel so I'd lov

• jamesfinlayson 5y ago
  Not sure - are Google running their servers with custom-made processors as well?
  • quantumofalpha 5y ago
    Kind of, I've seen some non-public Intel SKUs in the fleet but afaik they just had tweaked core counts and clocks for better perf/watt. They were also exploring non-intel architectures at some point, ppc especially but no significant prod deployment yet. And then I'm sure you've heard of special purpose hardware like TPUs
  • cube00 5y ago
    That's covered earlier (full quote): > Those patches implement various internal APIs (e.g. for Google Fibers), provide hardware support, add performance optimizations, and contain other "tweaks that are needed to run binaries that we use at Google". I'd assume if you're making allowances for custom-made processors at the kernel level you wouldn't also want that leaking into your user land binaries as well.
    • eru 5y ago
      Why not? They could have kernel accommodation for one feature, and user-land leakage for another. Also, for some features you need both kernel and userland cooperation. Just think of eg fuse or io_uring or mmap that are in the public kernel. You can surely imagine that Google might brew up similar features. (I used to work at Google. But I didn't have any special insider knowledge about their kernel stuff. Which is good, so I can't violate any lingering NDAs here with my speculation..)
  • jeffbee 5y ago
    A lot of it is just mundane details. Some of it leaks out in ABSL, for example some boring trivia is at: https://github.com/abseil/abseil-cpp/blob/master/absl/base/internal/sysinfo.cc#L303-L313 https://github.com/abseil/abseil-cpp/blob/master/absl/base/i... They also have for years had major patches to TCP which Linux refuses to adopt. Some details at: https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tcp-options-for-low-latency-00.pdf https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tc... These won't make binaries "not run" as such but they are necessary for the correct and efficient operation of large-scale distributed systems. QUIC, Snap/Pony, and user-space thread scheduling are strictly superior to what happens inside the kernel and their development makes the delta between prodkernel and upstream kernel less relevant over time. They also, collectively, make Linux itself less relevant.
    • pjmlp 5y ago
      Linux kernel is the most sucessful free beer UNIX clone, but it is by no means the only one, and if the trends on IoT and cloud infrastructure are any measure, in a couple of decades we might still live under POSIX monoculture to some extent, but Linux most likely won't be the kernel underneath.
  • H8crilA 5y ago
    Not CPUs but a lot of other components are completely custom, developed in-house. A well known example: TPUs. Google is one of the world's largest manufacturer of servers and other DC equipment, actually, it's just that none of that is for sale (though a lot is for rent :) ).
    • walrus01 5y ago
      Unless they've been building any factories I don't know about, their contractors are the manufacturers. I'm talking about companies based in Taiwan like Quanta, Compal, Clevo, and others. If you've ever been to a Computex Taipei you have seen the booths of the actual manufacturers of Google's hardware. https://en.wikipedia.org/wiki/Computex https://en.wikipedia.org/wiki/Computex
      • deleted 5y ago
        [deleted]
      • drivebycomment 5y ago
        This is a meaningless nitpicking. It is like nitpicking Apple is not a manufacturer of hardware since the actual assembly happens under some contractor. For all intent and purpose, and even technically, Google is the largest manufacturer of data center-sized supercomputers.
    • ars 5y ago
      See this for an early example: https://news.ycombinator.com/item?id=542531 https://news.ycombinator.com/item?id=542531
    • skunkworker 5y ago
      Like when that Pluto Switch got leaked back in 2012 https://www.wired.com/2012/09/pluto-switch/ https://www.wired.com/2012/09/pluto-switch/
  • eru 5y ago
    Some of them, yes! At least for machine learning. They also have custom silicon in their phones (and the phones run a Linux).
• yunohn 5y ago
  > Most of the world manages to run their binaries on a mainline kernel Most of the world doesn’t need to worry about their kernel - this is a good thing. But at FAANG scale, you inevitably need to make changes and optimizations. Eg: “Twitter has a kernel team!?” - https://danluu.com/in-house/ https://danluu.com/in-house/
  • twtrgrunt 5y ago
    I thought the kernel was pretty stable. My first team at twitter I think we ran into 2 or 3 kernel bugs that year. I couldn't believe it but for some problems the blaming the kernel is a real debugging step (and even the bios lol). I think I had two perf problems turn into upstream patches thanks to the kernel team.
    • johannes1234321 5y ago
      It's not only about bugs, but if you know your workload and your systems you can do some tricks a general purpose kernel can't do to tweak it for your needs and get a benefit here or there for your code.
  • pinerd3 5y ago
    that was a very interesting article; thanks for posting it :)
    • yunohn 5y ago
      Np, I saw it on HN last week :) Discussion: https://news.ycombinator.com/item?id=28691676 https://news.ycombinator.com/item?id=28691676
• bsder 5y ago
  In addition to hardware support, the article also mentioned adjustments to the perennial Linux pain in the asses that are the scheduler and the OOM killer.
• matheusmoreira 5y ago
  I remember reading an article about new system calls proposed by Google. It was a scheduler yield system call that allowed the current task to yield its remaining processing time to another task. It would presumably be used to make goroutines more performant. Not sure if the system call was accepted or if it still exists as Google specific kernel code. I can't seem to find the original article either...
  • jeffbee 5y ago
    I think you might be thinking of https://lwn.net/Articles/826860/ https://lwn.net/Articles/826860/
  • kccqzy 5y ago
    Here: https://youtu.be/KXuZi9aeGTw https://youtu.be/KXuZi9aeGTw Jump to the 15-minute mark for the three new system calls, switchto_wait, switchto_resume, switchto_switch.
• jdub 5y ago
  Ask a Googler about the size of their environment variables.
  • packetslave 5y ago
    ever wonder why linux supports command line arguments that are 2mb long?
    • yegle 5y ago
      Ever wondered absl flags library has a --flagfile special flag? https://abseil.io/docs/python/guides/flags#special-flags https://abseil.io/docs/python/guides/flags#special-flags
      • packetslave 5y ago
        I wonder if ABSL supports gzipped flags...
        • yegle 5y ago
          Probably not, since --flagfile accepts a list of filenames :-) https://github.com/abseil/abseil-cpp/blob/1ae9b71c474628d60eb251a3f62967fe64151bb2/absl/flags/parse.cc#L92 https://github.com/abseil/abseil-cpp/blob/1ae9b71c474628d60e...
    • matheusmoreira 5y ago
      Does Linux need explicit support for this? Command line arguments and environment variables are copied to the stack of the new process. I would expect the stack's size to be the limiting factor.
      • jeffbee 5y ago
        Before they relaxed it 15 years ago, Linux had a hard limit at 128KiB for args + env on x86. After the change it was variable up to 2MiB. https://github.com/torvalds/linux/commit/b6a2fea39318e43fee84fa7b0b90d68bed92d2ba https://github.com/torvalds/linux/commit/b6a2fea39318e43fee8...
• yegle 5y ago
  As an example, cgroup was initially developed in Google: https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Cgroups
  • catern 5y ago
    Yes, and it shows (it's notoriously poorly designed)
    • GolDDranks 5y ago
      Is Google notorious from poor design, or is there some other reason why cgroup being poorly designed would have to do it being designed by Google?
      • pjmlp 5y ago
        From Android point of view, poor design.
      • catern 5y ago
        When it comes to Linux kernel work, yes.
    • renewiltord 5y ago
      But it has the best design property of software: it exists and works in production.
      • pjmlp 5y ago
        That is what I tell myself when I get to review offshoring code.
  • menage 5y ago
    Cgroups is a great example of the problems faced here. The cgroups code/API that made it into the upstream kernel was the product of a couple of years of internal experimentation at Google into ways to do kernel-level resource isolation, and was pretty different from the approach that was used internally in production on a rather older kernel version. (A bit like Borg vs Kubernetes). And it still took a couple more years for cgroups to replace the original internal mechanisms, since the internal way worked OK and upgrading the kernel across so many machines was risky.
• geofft 5y ago
  Arguably, every line of code in the Linux kernel came from someone who couldn't run their binaries on mainline.
• kevincox 5y ago
  It is sometimes hard to imagine the scale of Google. If you can implement a task-switching mechanism with lower overhead and save 0.1% of CPU cycles across the fleet you can save many millions of dollars a year. So they have their own thread-switching mechanism which their binaries that can't run without. If you can improve process isolation in a way that allows you to pack 10% more tasks (by RAM or CPU) onto a single machine without affecting performance significantly you have just saved billions. So they have different isolation interfaces that Borg can't run without. Both of these are completely feasible savings and make custom binaries and a custom kernel completely with it.