14 ms·
Moving Google toward the mainline
- cube00 5y ago> contain other "tweaks that are needed to run binaries that we use at Google". Most of the world manages to run their binaries on a mainline kernel so I'd love to know what's so special in their binaries.
- jamesfinlayson 5y agoNot sure - are Google running their servers with custom-made processors as well?
- quantumofalpha 5y agoKind of, I've seen some non-public Intel SKUs in the fleet but afaik they just had tweaked core counts and clocks for better perf/watt. They were also exploring non-intel architectures at some point, ppc especially but no significant prod deployment yet. And then I'm sure you've heard of special purpose hardware like TPUs
- cube00 5y agoThat's covered earlier (full quote): > Those patches implement various internal APIs (e.g. for Google Fibers), provide hardware support, add performance optimizations, and contain other "tweaks that are needed to run binaries that we use at Google". I'd assume if you're making allowances for custom-made processors at the kernel level you wouldn't also want that leaking into your user land binaries as well.
- eru 5y agoWhy not? They could have kernel accommodation for one feature, and user-land leakage for another. Also, for some features you need both kernel and userland cooperation. Just think of eg fuse or io_uring or mmap that are in the public kernel. You can surely imagine that Google might brew up similar features. (I used to work at Google. But I didn't have any special insider knowledge about their kernel stuff. Which is good, so I can't violate any lingering NDAs here with my speculation..)
- jeffbee 5y agoA lot of it is just mundane details. Some of it leaks out in ABSL, for example some boring trivia is at: https://github.com/abseil/abseil-cpp/blob/master/absl/base/internal/sysinfo.cc#L303-L313 https://github.com/abseil/abseil-cpp/blob/master/absl/base/i... They also have for years had major patches to TCP which Linux refuses to adopt. Some details at: https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tcp-options-for-low-latency-00.pdf https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tc... These won't make binaries "not run" as such but they are necessary for the correct and efficient operation of large-scale distributed systems. QUIC, Snap/Pony, and user-space thread scheduling are strictly superior to what happens inside the kernel and their development makes the delta between prodkernel and upstream kernel less relevant over time. They also, collectively, make Linux itself less relevant.
- pjmlp 5y agoLinux kernel is the most sucessful free beer UNIX clone, but it is by no means the only one, and if the trends on IoT and cloud infrastructure are any measure, in a couple of decades we might still live under POSIX monoculture to some extent, but Linux most likely won't be the kernel underneath.
- H8crilA 5y agoNot CPUs but a lot of other components are completely custom, developed in-house. A well known example: TPUs. Google is one of the world's largest manufacturer of servers and other DC equipment, actually, it's just that none of that is for sale (though a lot is for rent :) ).
- walrus01 5y agoUnless they've been building any factories I don't know about, their contractors are the manufacturers. I'm talking about companies based in Taiwan like Quanta, Compal, Clevo, and others. If you've ever been to a Computex Taipei you have seen the booths of the actual manufacturers of Google's hardware. https://en.wikipedia.org/wiki/Computex https://en.wikipedia.org/wiki/Computex
- deleted 5y ago[deleted]
- drivebycomment 5y agoThis is a meaningless nitpicking. It is like nitpicking Apple is not a manufacturer of hardware since the actual assembly happens under some contractor. For all intent and purpose, and even technically, Google is the largest manufacturer of data center-sized supercomputers.
- ars 5y agoSee this for an early example: https://news.ycombinator.com/item?id=542531 https://news.ycombinator.com/item?id=542531
- skunkworker 5y agoLike when that Pluto Switch got leaked back in 2012 https://www.wired.com/2012/09/pluto-switch/ https://www.wired.com/2012/09/pluto-switch/
- eru 5y agoSome of them, yes! At least for machine learning. They also have custom silicon in their phones (and the phones run a Linux).
- yunohn 5y ago> Most of the world manages to run their binaries on a mainline kernel Most of the world doesn’t need to worry about their kernel - this is a good thing. But at FAANG scale, you inevitably need to make changes and optimizations. Eg: “Twitter has a kernel team!?” - https://danluu.com/in-house/ https://danluu.com/in-house/
- twtrgrunt 5y agoI thought the kernel was pretty stable. My first team at twitter I think we ran into 2 or 3 kernel bugs that year. I couldn't believe it but for some problems the blaming the kernel is a real debugging step (and even the bios lol). I think I had two perf problems turn into upstream patches thanks to the kernel team.
- johannes1234321 5y agoIt's not only about bugs, but if you know your workload and your systems you can do some tricks a general purpose kernel can't do to tweak it for your needs and get a benefit here or there for your code.
- pinerd3 5y agothat was a very interesting article; thanks for posting it :)
- yunohn 5y agoNp, I saw it on HN last week :) Discussion: https://news.ycombinator.com/item?id=28691676 https://news.ycombinator.com/item?id=28691676
- bsder 5y agoIn addition to hardware support, the article also mentioned adjustments to the perennial Linux pain in the asses that are the scheduler and the OOM killer.
- matheusmoreira 5y agoI remember reading an article about new system calls proposed by Google. It was a scheduler yield system call that allowed the current task to yield its remaining processing time to another task. It would presumably be used to make goroutines more performant. Not sure if the system call was accepted or if it still exists as Google specific kernel code. I can't seem to find the original article either...
- jdub 5y agoAsk a Googler about the size of their environment variables.
- packetslave 5y agoever wonder why linux supports command line arguments that are 2mb long?
- yegle 5y agoEver wondered absl flags library has a --flagfile special flag? https://abseil.io/docs/python/guides/flags#special-flags https://abseil.io/docs/python/guides/flags#special-flags
- packetslave 5y agoI wonder if ABSL supports gzipped flags...
- yegle 5y agoProbably not, since --flagfile accepts a list of filenames :-) https://github.com/abseil/abseil-cpp/blob/1ae9b71c474628d60eb251a3f62967fe64151bb2/absl/flags/parse.cc#L92 https://github.com/abseil/abseil-cpp/blob/1ae9b71c474628d60e...
- matheusmoreira 5y agoDoes Linux need explicit support for this? Command line arguments and environment variables are copied to the stack of the new process. I would expect the stack's size to be the limiting factor.
- jeffbee 5y agoBefore they relaxed it 15 years ago, Linux had a hard limit at 128KiB for args + env on x86. After the change it was variable up to 2MiB. https://github.com/torvalds/linux/commit/b6a2fea39318e43fee84fa7b0b90d68bed92d2ba https://github.com/torvalds/linux/commit/b6a2fea39318e43fee8...
- yegle 5y agoAs an example, cgroup was initially developed in Google: https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Cgroups
- catern 5y agoYes, and it shows (it's notoriously poorly designed)
- GolDDranks 5y agoIs Google notorious from poor design, or is there some other reason why cgroup being poorly designed would have to do it being designed by Google?
- renewiltord 5y agoBut it has the best design property of software: it exists and works in production.
- pjmlp 5y agoThat is what I tell myself when I get to review offshoring code.
- menage 5y agoCgroups is a great example of the problems faced here. The cgroups code/API that made it into the upstream kernel was the product of a couple of years of internal experimentation at Google into ways to do kernel-level resource isolation, and was pretty different from the approach that was used internally in production on a rather older kernel version. (A bit like Borg vs Kubernetes). And it still took a couple more years for cgroups to replace the original internal mechanisms, since the internal way worked OK and upgrading the kernel across so many machines was risky.
- geofft 5y agoArguably, every line of code in the Linux kernel came from someone who couldn't run their binaries on mainline.
- kevincox 5y agoIt is sometimes hard to imagine the scale of Google. If you can implement a task-switching mechanism with lower overhead and save 0.1% of CPU cycles across the fleet you can save many millions of dollars a year. So they have their own thread-switching mechanism which their binaries that can't run without. If you can improve process isolation in a way that allows you to pack 10% more tasks (by RAM or CPU) onto a single machine without affecting performance significantly you have just saved billions. So they have different isolation interfaces that Borg can't run without. Both of these are completely feasible savings and make custom binaries and a custom kernel completely with it.
- sudarshnachakra 5y agoA genuine question. Why is that Google uses a custom kernel (with patches that could not be mainlined) when the below options are available. 1. Compiling the kernel with custom knobs 2. Write the custom code as Linux modules
- extr 5y ago[deleted]
- brantonb 5y ago>Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- ndesaulniers 5y agoI'd imagine drivers for lots of custom hardware that will never leave the datacenters.
- CameronNemo 5y agoBut drivers for custom hw are easy to rebase. They are quite self contained. It is when you get into kernel internals that rebasing continuously becomes a challenge. Some parts of the core Linux kernel don't change often, but I imagine many other parts see significant churn.
- dboreham 5y agoBecause originally they were (way) ahead of the mainline. Article doesn't actually say this fwiw. Nowadays mainline is mostly caught up, but it's hard to rebase (article does say this).
- danobi 5y agoBecause you can’t add a feature with a knob. And not all changes can be made in a module (eg core kernel). And even if they could, kernel internals are not stable and you would still need to rebase your out of tree modules.
- dboreham 5y agoThis kind of "feature branch composability" can be challenging, because the code on the branches turns out to not be entirely orthogonal.
- hinkley 5y agoThere’s a whole nasty class of bugs introduced by bad merge resolution. Mark was a senior front end Dev who was bad at complex logic. He sat near me and I took pity on him, helped him out to help us out. Steve was smarter and more experienced but had attention to detail problems and did not understand git. He should have been one of my lieutenants but I could not trust him and often had to help him do branch surgery. Steve did not like this arrangement but it couldn’t be helped. He got cranky about it and went after Mark. I can’t prove he was coming at me sideways, but it was curious. One day Steve claims Mark broke our shit. Here’s this line with his name on it. It even looked like a Markism. But the thing was, I reviewed that code. I remember being happy Mark got it right on the first try. This was not the code I merged but git says it is. Da fuq? So I start bisecting and looking at branches and sure enough, Steve screwed a three way merge, again, and blame treated the change as if Mark wrote it. Thanks, you stupid git.
- ggm 5y agoIs this part of the google monrepo? Not that it matters, but I wondered how far monrepo went, inside the tent. I hate to be a pessimist, and I may be simply ignorant of some realities, but it FEELS like this is setting an impossibly high goal. I think there will turn out to be something significant in 9000 variances, which cannot simply be uplifted into mainstream, and which turn out to be hard to maintain as a continual rebase into the emerging mainstream. I agree that 40 sounds low, but probably is a good indication it can work. 1% is within view, if they maintained pace so even on a 1% if the rebase cost can be handled, they can converge but that assumes there isn't a showbuster in the 9000.
- xyzzyz 5y agoFrom what I remember, prodkernel actually was outside google3 (the monorepo), and has been sitting in a git repository.
- anonygler 5y agoBuilds are all derived from the monorepo, but git makes sense for development as they have to constantly merge upstream git commits.
- mvgoogler 5y agoProdkernel is not in the monorepo - it's in a separate git repo and uses it's own systems for building and qualifying new releases. There is a system for using git* in conjunction with the monorepo that some developers use, but that system has nothing to do with the prodkernel. * there is also a similar system for using mecurial with the monorepo that some folks love.
- drewg123 5y agoNo. At least when I was there, the linux kernel was in git, and reviews were done in Gerrit. The monorepo (google3) and its review tools were used for most other projects.
- 5y ago
- CameronNemo 5y agoI really hope they try to tackle OOM upstream. Every solution I've seen is "run this daemon that tries to beat the kernel OOM to the punch". Seems inelegant.
- hinkley 5y agoNodejs uses this and I hate it. Tune it wrong and you get no logs, or only in a spot nobody looks. Also you end up restarting sooner than you need because you have to plan for worst case where all processes flirt with the high water mark all the time, so you restart at 1.1/n even though 1.2/n might be okay.
- esgwpl 5y ago>"run this daemon that tries to beat the kernel OOM to the punch" Considering that the kernel OOM killer tends to be way too late in doing its thing I don't see how this is inelegant, maybe there's a reason you can't just have the kernel kill processes earlier in the face of memory pressure.
- ladyanita22 5y agoThe kernel OOM is just plain broken. I can't understand how can it be possible that, on Windows, whenever I run out of RAM the OS kills whatever process is consuming too much and the computer keeps running flawlessly. However, on Linux, my computer just... freezes. It freezes and stops responding. Not even the mouse moves. Having to use a userspace OOM is the most inelegant thing I've seen. So I need to have 2 OOMs so that the good one can beat the bad one? It's so redundant, it's plain stupid. How come NOBODY is doing anything. If I knew c++ I would for sure send a patch.
- School-Cotton 5y agoThe Linux kernel is written in C, not C++.
- ladyanita22 5y ago
- encryptluks2 5y agoIt seems like they could work with upstream to figure out a way to restructure source files and add commented markup to make the process a lot easier. Not that other people should do Google's work, but such changes could be beneficial to multiple people and organizations that need to build custom kernels, like realtime Linux, Xen, and hardened.
- sharmin123 5y agoThe Effective Positive Parenting Tips And Avoid Parenting Challenges: https://www.hackerslist.co/the-effective-positive-parenting-tips-and-avoid-parenting-challenges/ https://www.hackerslist.co/the-effective-positive-parenting-...
- gorgoiler 5y agoYou could have editorialized this title to: Rebasing a Linux Kernel Branch that’s Two Years Old and 9k Commits Deep …but you would have had thousands of us running to the bathroom to revisit our last meal. What a Herculean effort by these two engineers. When your role is to track down fellow employees from completely different teams and crack the, er, carrot to get them to update their patch from two years ago, you can get very tired very quickly. I’m sure the hardware driver modules are fairly self contained but things like the userspace QoS patch could be pretty hairy. The kind of thing where, for upstream, you’d spend 90% of your time landing a framework just to even be able to implement your feature with the final 10% patch. Oof. Bravo.
- lt 5y agoAs far as I understood, your editorialized title is what they've been doing every two years since ever. A possible better editorialized title would be Creating a parallel kernel so we can continually rebase commits instead of every two years
- Too 5y agoDon't forgot the part where the author of the patch left the company a year ago, or even worse the entire team got dissolved/reorganized, which tends to happen quite regularly in a bigco. No amount of carrots will help there I'm afraid. Herculean effort indeed.
- menage 5y ago> The kind of thing where, for upstream, you’d spend 90% of your time landing a framework just to even be able to implement your feature with the final 10% patch. I feel this was a fair description of CGroups.
- fsckboy 5y agothis is, pure and simple, technical debt. Google borrowed for proprietary reasons; meanwhile the people who maintain and improve the kernel and give away the fruits for free have produced something that google wants so they'll merge, again for FAANG proprietary reasons. I'm not losing sleep over it, hell, I paid for it already with my privacy.
- tut-urut-utut 5y agoLet's not pretend that kernel developers still don't get paid for their kernel work. The majority of them are employed by corporations that use Linux or are sponsored by other means. Maybe it was so in the past, but I doubt that any significant contributions nowadays come as a free (as in beer) work.
- 5e92cb50239222b 5y agoThat's pretty much how it is. https://lwn.net/Articles/867540/ https://lwn.net/Articles/867540/ https://lwn.net/Articles/860989/ https://lwn.net/Articles/860989/ https://lwn.net/Articles/853039/ https://lwn.net/Articles/853039/
- deleted 5y ago[deleted]
- Jensson 5y agoYou do realize that almost every big company pays money to the Linux foundation and that Linus Torvalds is paid 10 million dollars a year thanks to that? They still get it very cheaply, but that is because the Linux team is very efficient and not because they aren't paid.
- chinathrow 5y ago> Linus Torvalds is paid 10 million dollars a year thanks to that Any source for this claim?
- fcsp 5y agoIs their fork source code available somewhere publicly?
- kevincox 5y agoI'm pretty sure that it isn't.
- fcsp 5y agoThat's sad. How does that work with the GPL license of the kernel?
- kevincox 5y agoThe compiled kernel isn't distributed so there is no need to distribute source. Note that prodkernel is different than the Android kernel (where they do, and are required to share the source by GPL). To be honest I don't think it is a major loss. IIUC most features are either only interesting internally, or are submitted upstream to reduce maintenance cost. Google's "secret sauce" isn't how great their kernel is, so I am not aware of anything that is held back for competitive advantage other than maybe some drivers for custom hardware. So while it would be nice to see all of these patches collected into a "live" snapshot of their kernel it probably wouldn't be that much more helpful (if at all) to a third-party.
- knorker 5y agoWhat's that userspace qos thing? You can already set DSCP, so is this more like setting the actual shaping/policing policies for your sockets?
- hinkley 5y agoWould that be avoiding system call round trip overhead perhaps? I wonder how some of these tools will change due to progress on io_uring and similar idioms. Batching system calls to amortize the cost.
- knorker 5y agoHmm, seems unlikely, since at least for TCP this is just one syscall per connection. Syscalls are not that expensive.
- jeffbee 5y agoI don't know if this is what they were driving at but there are tons of useful things that you can't do in the sockets API like selectively up- or down-grading DSCP for a subset of TCP ACKs. Linux also does not come out of the box with DSCP reflection for the SYN+ACK, or at least it didn't until 5.10. Google prodkernel had this feature for at least a decade prior. https://lwn.net/Articles/831194/ https://lwn.net/Articles/831194/
- topbanana 5y agoThe amount of time and effort expended on this must be immense. Would it have been feasible at the time to sponsor having these changes pushed into mainline?
- hinkley 5y agoWhat’s really going to cook your noodle is thinking about Red Hat. They were maintaining long running forks when google was still in diapers.
- kevincox 5y agoMany if not most of these patches have been attempted to be mainlined. Either they were not considered of acceptable quality, considered too similar to other features (ex: Binder in Android) or rejected for other reasons (ex: Google's Fibers and SwitchTo). If you have a patch that saves significant resources, improves important performance metrics or unblocks hardware that does those things you don't care if upstream will take the patches or wait for them to decide if they will. You simply start using the patches ASAP and reap the benefits, then you continue trying to upstream the patches to reduce the maintenance burden.
- drewg123 5y agoWe did this with FreeBSD several years ago at Netflix. We used to run heavily modified -stable kernels. Our tree was ahead of FreeBSD-current in a lot of ways (TCP, ktls, VM) but way behind in other ways. Every year or two we'd rebase to a new -stable version, and that would be months of work. Often, there would be some bug introduced upstream many months prior that took weeks to track down because there was just so much to bisect through. Because our tree was based on -stable, pushing changes upstream often involved porting them to a substantially different codebase in -current, and being unable to test the ported code at scale in production. Since we've moved to tracking -current, we rebase every few weeks. We catch any newly introduced bugs from upstream almost immediately, since there are only a few weeks of changes, not a few years, to sort through. Its far easier to contribute code upstream. In fact, we do most development upstream and wait until the next rebase to pick our changes up (or cherry pick them if it is something urgent). All in all, life is much, much, much better on the bleeding edge.
- hinkley 5y agoDo you run all service on the new kernel or is there a roll out process where some services are canaries? I have one service that is low traffic that I use to test out new base images and core api changes. It’s easy to generate synthetic traffic to see how it behaves, as a lot of its traffic comes from the build process of more complex services.
- drewg123 5y agoAll machines (can) run the same services, and we roll an entirely new self contained OS image for every update. That goes through multiple expanding levels of canaries prior to a fleet wide rollout. Bugs that escape the canaries are quite rare. One of the last ones I recall was a release that contained a firmware update for a NIC that had a super rare bug that would lock up the NIC, and we only started to see it when a sizable percent of the fleet was running that firmware.
- capableweb 5y ago> All in all, life is much, much, much better on the bleeding edge. Same feeling here, but on a personal level (meaning, using a desktop computer for professional work). Started using Ubuntu many years ago, ran updates maybe once every month. Every now and then something breaks, but always unsure of what/how/when, so digging through things that changed always took long time. Started using Arch Linux and few years ago and also started doing updates at least once per day. Now it's way easier to discover what particular change broke something, as the releases are way smaller. I attribute this more to how often I do the updates now VS before, and less to which OS you use. Same can be achieved with Debian Testing et al.
- yalok 5y agois there a video of their talk, or slides? Seems to be behind a paywall on OSSNA site...