14 ms·

How Google got to rolling Linux releases for Desktops

emptysongglass 4y ago
The more important question is where is this magical Google-blessed rolling release Debian available to download? We've all heard rumors of this internal Google distro and what I'm reading is their intent to give back. You can use Microsoft's CBL-Mariner today. Where is Google's?
- AdmiralAsshat 4y ago
  Why yes, they could share Goobuntu with the world. But they don't have to, and so they won't.
  - leoh 4y ago
    It’s really not that interesting
- etbusch 4y ago
  It seems like they are just using Debian testing with periodic version freezes. It's very likely not all that different from just running testing.
- brandon 4y ago
  Most of the stuff that makes the distribution different from Debian Testing is tightly coupled with Google's internal infrastructure and generally uninteresting (or unuseful) to users writ large. Just run Debian!
- qmarchi 4y ago
  If you want to get really technical, there's not much difference between Google's gLinux and Debian testing. Notably, our security, provisioning, and tooling is installed, and that's about it. Nothing that's particularly useful outside of the corp bubble. Disc: Googler, not in CorpEng.
  - ryanobjc 4y ago
    Not to mention many of those things don't work outside the rarified environment that is a fully managed VLAN. Regarding "security", remember plenty of security is things like auditing, remote logs, ability of remote admins to control machines, etc, etc. It's ultimately all boring, but essential, large fleet management stuff.
- POPOSYS 4y ago
  - DiggyJohnson 4y ago
    I have never seen a gender-based criticism of the fragmentation of Linux distros, that’s for sure…
Barrin92 4y ago
personally I'm a pretty big fan of opensuse tumbleweed for a rolling distribution because they ship fully tested snapshots each day rather than individual packages. If I understand the article correctly Google's Sieve is similar to the Opensuse Build Service in that regard. It seems like they went for Debian because of the proximity to Ubuntu but I wonder if they ever considered Suse, the ecosystem is pretty good.
- themodelplumber 4y ago
  Thanks for your comment. I have been thinking about Tumbleweed or Manjaro to try out a rolling release. Not sure yet which to choose, as they seem quite different in a lot of ways.
- compiler-guy 4y ago
  The upgrade-in-place plan is made more difficult the more distant from Ubuntu you get. Suse has a great ecosystem, but the transition costs would have been quite a bit higher.
- abirch 4y ago
  I liked Suse before Novell acquired it. Is it still good?
  - Arnavion 4y ago
    Yes. I've been using it for over a decade and it's still great. Like Fedora it's leading the charge in modernization (btrfs, /usr/etc, snapper, Yast, etc) and like Arch it has a big set of up-to-date packages, except they're in the main repo instead of an untrusted AUR.
    - nyanpasu64 4y ago
      You need third party repositories if you want full ffmpeg with h.264 support, and since these are binary (IIRC) repos, each third-party package is often either compatible with the latest or older snapshot, or tumbleweed, but often not both (unlike Arch's AUR which only has one distro to target, though it's a moving target).
lynguist 4y ago
Can I ask the following as this what Google did is literally what I try to achieve on a Raspberry Pi: I have configured unattended-upgrades (the package from apt) on a Raspberry Pi on Raspbian 10, but no security updates seem to ever be released. Is there a gotcha I’m missing resp. is there a better way of configuring an always-on computer and keep it secure?
- pertbert 4y ago
  The article mentions: > We build packages in package groups, to take into account separate packages that need to be upgraded together. Once the whole group has been built, we run a virtualized test suite to make sure none of our core components and developer workflows are broken. Each group is tested separately with a full system installation, boot and local test suite run on that version of the operating system. > We then proceed to carefully guide this release to the fleet utilizing SRE principles like incremental canarying and monitoring the fleet health.
- StillBored 4y ago
  Raspbian, or raspberry pi os? The latter is much closer to debian AFAIK, there is a 64-bit version/etc. But if your using it for server tasks, your probably better off actually running Debian. Although if your looking for something that is closer to a rolling release, you could just pick one of the distro's that have a release policy that strikes your fancy WRT how they roll out new packages/etc. Most distro's have rpi specific images, or you just install their generic arm64 images on top of something like the PFTF firmware. Put another way, raspberry pi os, has a fairly dated kernel/etc but works well with all the hardware on the machine for desktop purposes. If your using it as a server, its not a good choice because they frequently don't have useful things turned on (various filesystems, networking options, RAID levels, etc). The core storage/networking/cpu/etc type of things have been working quite well in mainline/etc for a couple years now, so that has trickled down into all these distros.
- sillystuff 4y ago
  It sounds like you missed a step. You need to create an apt conf file that runs unattended-upgrades (in addition to the configuration you already did in /etc/apt/apt.conf.d/50unattended-upgrades). There are a lot of options (see /usr/lib/apt/apt.systemd.daily for a description of options; the manpages are lacking), but just running: dpkg-reconfigure unattended-upgrades will create a minimal config file, '/etc/apt/apt.conf.d/20auto-upgrades' that may be all you need. But, you might consider adding something like below to cleanup downloaded package files: // Auto cleanup archives APT::Periodic::MaxAge "10"; // Setting minage too is a good idea to prevent race conditions APT::Periodic::MinAge "8"; Or, if you prefer (but, this will delete all archives present when it triggers and not just old ones like above): APT::Periodic::AutocleanInterval "10"; (Note: I'm assuming Raspbian is the same as the upstream Debian; I've never used Raspbian)
glouwbug 4y ago
Just use Arch
- whateveracct 4y ago
  Or NixOS unstable hehe
  - zamalek 4y ago
    The nice thing with flake-based NixOS is that it's trivial to cherry-pick unstable onto a stable base. I do a bunch of that in my nixconfigs: https://gitlab.com/jcdickinson/nix https://gitlab.com/jcdickinson/nix
    - whateveracct 4y ago
      Thanks for sharing. I'm new to flakes, so seeing that idiom of having a separate nixpkgs-unstable arg propagated around set off a lightbulb!
- Pr0ject217 4y ago
  EndeavourOS here (arch-based). Love it.
  - glouwbug 4y ago
    Nothing like a good ol' 9 month late pacman -Syyu
    - Shared404 4y ago
      I dunno, a late xbps-install -Su may give it a run for it's money
- dinvlad 4y ago
  I wanted to give it a try, but SecureBoot is refusing to be set up correctly, despite following the manual '( (I suspect the problem is with NVidia drivers..)
dxneezal 4y ago
I tried to search more regarding Rodete then found this[1] > … was to get off of Ubuntu's more classic LTS release cycle, and move to the freely-rolling waters of Debian Unstable. An underpublicized but very real reason was that the Google Linux team had a number of Debian Developers on it, who reasonably wanted to be closer to their primary love and affiliation :blink :blink 1: https://www.reddit.com/r/debian/comments/j4liv4/whats_going_on_with_glinux/ https://www.reddit.com/r/debian/comments/j4liv4/whats_going_...
dredmorbius 4y ago
Alt link: https://archive.ph/42l8J https://archive.ph/42l8J
sytse 4y ago
Cool article. My two takeaways: 1) If your deployment process hurts do it it more often (small releases frequently are easier than few big bang releases. 2) Don't commit to being x days behind upstream, if there is a big workload on your team (upgrading, vacations, etc.) you have the flexibility to delay upgrades and reduce the stream of incoming issues.
- baby 4y ago
  This is true for your codebase and your dependencies as well.
  - erik_seaberg 4y ago
    This, remaining on a known-good release is a form of technical debt to use judiciously and pay down when more convenient. Beta testing everything immediately can be a favor to the community, but it’s probably not your main job.
    - yazaddaruvala 4y ago
      Shouldn’t need to be anyones job. Automate: Daily chron: - Branch the repo, auto-update one dependency (ideally a smarter way to batch up groups) - Run CI - Auto-merge commit if CI passes, else discard commit. - Loop to create a new branch for the next dependency waiting to be auto-updated. Otherwise, not having the right testing or CI is technical debt like the grand parent commenter suggests.
    - jillesvangurp 4y ago
      Actually, it's the opposite in my experience. Beta testing is typically done when most software is released. Basically, at that point your technical debt is all the known & fixed issues that you still have, the features you don't have access to and can't benefit from, the performance issues you still have, the newly deprecated APIs that you are still using, etc. Opting into all that without good reason is a bad idea. If you are afraid things will break, the best way to find out is to just try it. Worst case you have to roll back a change. But at least then you know and you can plan a fix. IMHO anything that isn't on a current version should have a documented reason why that is or be fixed ASAP. I rarely see this on projects. Usually that means nobody cared enough to try which is just a form of negligence. On most of my projects I update dependencies whenever I can. On my older projects, I don't even touch any source code until after I update dependencies whenever I do maintenance on them. Typically either nothing breaks, I need to do some minor fixes, or I need to temporarily roll back some dependency to plan some bigger change. The thing is, if a new version is going to require any kind of non trivial work, I want to know about this as early as possible; especially if it is a lot of work. If you wait a year, you are looking at a lot of unknown work, which is basically technical debt you did not even know you had. I don't like that kind of uncertainty. Mostly staying on top of things minimizes the work you actually have to do. And you get to benefit from all the fixes early. A lot of projects I join are hopelessly outdated. It's usually the first thing I fix. It's rare to find documented reasons why a particular thing can't or shouldn't be updated. If it's not documented, I'm just going to go ahead and fix it. In case it doesn't work, I'll document it.
TheRealDunkirk 4y ago
I always thought creating your own rolling distro with Gentoo was precisely the best case use of portage, but I’ve never seen anyone do it. I must be missing something.
- tyingq 4y ago
  Guessing because a fair amount of stuff is tested, packaged, or unintentionally tied to how either Debian, Ubuntu or RedHat does things (or places things, etc). So anything outside that universe doesn't get popular.
- euank 4y ago
  Coincidentally, Google has sorta done that too! ChromiumOS/ChromeOS is built using portage, and is certainly rolling release. It's not really a traditional linux distro, but it's something.
  - antoncohen 4y ago
    Container-Optimized OS is based on Chromium OS and is more of a traditional Linux, in the sense that is runs on servers. https://cloud.google.com/container-optimized-os/docs https://cloud.google.com/container-optimized-os/docs
    - eliaspro 4y ago
      ...and so was CoreOS before the equally named company was acquired by RedHat.
      - kaeso 4y ago
        In a funny twist of events, the author of the grandparent comment above yours was indeed building that OS (Container Linux) at CoreOS :)
YPPH 4y ago
Out of curiosity, why would a Googler elect to use a Linux based workstation as opposed to a macOS workstation? Are they testing Linux binaries? Developing kernel extensions? Linux build tools? macOS offers such a superior desktop experience I can't imagine why else you'd ask an employer offering you a macOS machine to substitute it with a Linux one.
- vinkelhake 4y ago
  The average Googler doesn't get to pick the OS that their workstation runs. It's going to be Linux. The typical work setup is a browser, some kind of editor and terminals - which is something that works well in Linux.
  - qmarchi 4y ago
    It depends on the team, and the individual. Generally, a Googler does actually get to pick their OS, it's part of the onboarding process that every Noogler goes through, and you're given 30 days from your start to swap around and test the other platforms (gLinux, macOS, gWindows, or Chrome OS).
    - vinkelhake 4y ago
      I was under the impression that unless you have an actual business need that calls for something other than gLinux, then it's going to be gLinux. Like a Chromium dev who primarily works on the Windows side of Chrome obviously has a need for it. I'll fully admit that my onboarding experience is quite out of date since I've been there for a decade. When I joined I didn't really have a choice for workstation since I only worked on server side stuff. Sounds like things have changed.
      - esprehn 4y ago
        No they're the same. Pretty much all google3 work is done on Linux, but folks usually get 2 machines: a laptop you pick the OS and the workstation (or cloudtop) which is Linux unless there's a special need like chrome eng. Lots of folks do stuff like use Chrome OS + cider + cloudtop though.
colordrops 4y ago
I'm surprised no one has mentioned NixOS. Easy rolling forward or back, with config, system snapshots, and cached binaries. Can roll individual packages or the whole system very easily.
- pbamotra 4y ago
  From the article, > Rolling releases with Linux distributions today are getting more common (Arch Linux, NixOS).
  - uncletaco 4y ago
    NixOS users are socialized to not bother reading
    - TomK32 4y ago
      Can't roll back those HN comments though...
      - djbusby 4y ago
        Can edit and/or delete within some time window.
    - colordrops 4y ago
      I'm talking about the comments.
    - SuperSandro2000 4y ago
      Btw I am using NixOS
pmontra 4y ago
Does anybody know which laptops is Google using to run their Linux distro on?
- smueller1234 4y ago
  Carbon X1's are common. It's not exclusively those though.
akdor1154 4y ago
Jesus.. here i am in my BigCorp being disallowed to dev with Linux because too much mandated corporate crapware doesn't work. I dream of the day my employer provides an in-house distro for me.
- HyperSane 4y ago
  WSL2 works really well.
  - p_l 4y ago
    As part of the crapware image process, WSL (1&2) is often blocked
  - happymellon 4y ago
    Unlikely to work in any corporate environment that I've witnessed. This is why so many devs I know moved to Mac's. Not because Mac's are necessary that great, who cares about battery life when it just sits on a desk, but because it's the closest you can get to Linux in a corporate environment. I would take native Linux on an ThinkPad as first option, but denied that I would go for Mac over Windows. I don't find WSL fixed all the other Windows problems such as lack of support for multiple desktops. They exist but the implementation is terrible.
    - surban 4y ago
      Windows 11 natively supports multiple desktops.
      - happymellon 4y ago
        Is it better than 7 or 10? As those "supported" multiple desktops, but was implemented in such a bad way. Gnome and KDE are so much smoother when it comes to these workflows.
    - SuperSandro2000 4y ago
      Mac's are closer to BSD than Linux and suck for Linux oriented development. Brew is more ductape than a solution and you basically need to replace all GNU tools since they are horrible out of date.
deleted 4y ago
[deleted]
walrus01 4y ago
regular debian testing, unless you are doing something really esoteric, is really quite stable for desktop use. such as you might get if you installed the basic barebones debian CLI setup totaling about 1GB disk space occupied and then used apt to install xorg and xfce4 or any other window manager. debian-stable is extremely conservative in packages and stuff is often several years behind.
- fulafel 4y ago
  What about security updates? They might come slowly to unstable (since security team doesn't concern themselves with unstable) and might be held up arbitrarily long before trickling down to testing, no?
- deleted 4y ago
  [deleted]
wiseowise 4y ago
Why not ChromeOS?
- denysvitali 4y ago
  They use ChromeOS as well, this article focuses on those people who want to use a mainstream Linux Distro at Google.
  - mjg59 4y ago
    Pretty much everyone has a gLinux workstation or logs into a gLinux VM - they were, traditionally, the only devices that had access to the main source repository, so it's not so much about whether people want to or not.
    - Morgawr 4y ago
      Googler and ChromeOS engineer here. Opinions are my own, etc etc. This has changed/is changing. It heavily depends on your team (especially if you're not in engineering, I don't think most non-engineers would have a gLinux machine or a workstation out of the box). We've developed tools for crostini that would let people access a gLinux environment, I don't know how much I can talk about it so I won't go into details (although it's nothing super secret either), but the bottom line is that more and more engineers these days are able to do 100% of their job on a ChromeOS machine, including dealing with google3 stuff.
      - saagarjha 4y ago
        I mean, you can always use the web-based Cider to do your work. It's not great, but it gets the job done.
- alpb 4y ago
  Likely because these are not end-user machines but developer machines. Most Google engineers don't checkout/write/build/test/run code on their machines (they either do these tasks through a browser or SSH to another machine). I believe gLinux can be installed on trusted machines that you connect over SSH as well as client laptops (where you can't checkout code etc). That said many engineers/non-engineers at Google use ChromeOS on their client laptops (same with macOS).
raffraffraff 4y ago
Glad to see that they're planning on contributing more to the upstream. But considering the size and budget of Google, and the amount of free work they're getting from the Debian community, it would be utterly irresponsible not to. Not blaming the team that's doing this good work though - I'm sure they're working hard enough as it is.
- ryanobjc 4y ago
  I think there's a big misunderstanding as to what exactly gLinux _IS_. Nearly every large company I've worked at worked hard to standardize the dev environment. Ensuring the right set of dev tools (and more) is available on every machine. It's not as simple as 'apt-get install build-essential' and you're done. For example many large companies use perforce, which also has a licensing requirement. Google in particular has a large set of custom binaries, remote file systems, unified login (think krb but maybe not exactly that), unified management, etc, etc. That is what gLinux is. This isn't something you're going to want or need. In fact it doesn't even run outside of the Google hard wired VLANs really (for setup). There are a LOT of net benefits for much of google to being on gLinux. It means, for example, that nearly every product works on Linux - google docs, spreadsheet, meet, etc. That's a huge win. Not only that, they are on par with Windows or Mac support (thanks browser as a platform).
  - Zhyl 4y ago
    While this is true, I think a consumer gLinux (i.e. a Debian spin with support from Google) would be more appealing to Vendors than Ubuntu and similar just because of brand recognition. I think Google had the opportunity to accidentally displace some of the Windows monopoly, not by doing a land grab a la ChromeOS or Android, but instead by just releasing a version of something they use themselves which would be appealing to like minded developers and companies that would follow Google's 'stamp of approval'.
- baby 4y ago
  Open source = no strings attached. I don’t get the shaming of big users of open source. (And if we really want to get there: chrome, android, angular, etc.)
pabs3 4y ago
I'm using Debian testing for years with unattended-upgrades upgrading everything 4 times daily (once for each Debian repository update). Not at Google. AMA!
- encryptluks2 4y ago
  Switch to Arch for God's sake
- yasinaydin 4y ago
  Have you ever used/considered using another rolling release? If yes, how would you compare them to Debian testing? Also, are there any parts of Debian testing that is not rolling (i.e. OS version number or some package versions like kernel or Node.js)?
  - pabs3 4y ago
    I've always used Debian since I discovered it via Knoppix (which was my first Linux distro, Cygwin was my first Unix distro), I initially used Debian unstable but downgraded to testing some years later as upgrading testing often is smoother than unstable. I've never considered even trying any other distro. Everything is rolling, except unmaintained software, that just gets removed from the distro when other faster rolling things change incompatibly.
- RealStickman_ 4y ago
  Why 4 times daily and do you do that while you're working? How do you deal with packages being broken until the next reboot if you do updates during the day? I don't even update my Arch boxes that regularly and always do a reboot afterwards.
  - pabs3 4y ago
    The Debian archive is updated 4 times daily, so I update my system after each Debian update. I do that no matter whether I am working or not. Packages almost always aren't broken in Debian testing but if they were, a reboot would not fix them anyway. I restart processes instead of rebooting (using the needrestart/needrestart-session tools). I only reboot after Linux kernel image/module updates or microcode updates, because Debian cannot yet live-update those components.
baby 4y ago
Interesting post! I’ve been thinking about trying Linux with a rolling distrib. They mention NixOS, do people here have an opinion on that one?
- SuperSandro2000 4y ago
  NixOS is programmed in a functional programming language and aims to be fully pure and reproducible. Using it since over a year now and my entire hackspace runs on it. Can highly recommend it.
- Jedd 4y ago
  Why not Debian testing branch, as per TFA?
- funcDropShadow 4y ago
  I am not sure I would call NixOs a rolling release distribution. It has releases every 6 month like Ubuntu, but it does not have LTS releases. And due to the relatively small community only the latest release is supported. I am doing software development on a desktop workstation, that followed all releases since the start of 2018. And overall it has been rock solid. If an update, either a new release or a config change of mine, breaks something important, I reboot with the previous setup, and I can continue to work until I have time to investigate what is going on. I've used that perhaps 5 times since 2018. But when I used it, it was absolutely critical. But expect a steep learning curve.
  - kevincox 4y ago
    nixos-unstable is a rolling release. I use it daily and on all of my servers.
- kevincox 4y ago
  I love NixOS. I would 100% recommend it for servers and probably 70% recommend it for desktop usage. Feel free to ask me anything but instead of repeating past points I'll just link some past writings: - https://kevincox.ca/2015/12/13/nixos-managed-system/ https://kevincox.ca/2015/12/13/nixos-managed-system/ - https://kevincox.ca/2020/09/06/switching-to-desktop-nixos/ https://kevincox.ca/2020/09/06/switching-to-desktop-nixos/ - https://kevincox.ca/2021/05/06/workstation-install-with-nixos/ https://kevincox.ca/2021/05/06/workstation-install-with-nixo...
Wronnay 4y ago
Rolling Releases are basically my main reason why I choose Arch Linux over Debian. It would be nice, if Google and Debian could team up together to bring a Debian Version with rolling releases. Imagine the stability of Debian packages together with the easiness of upgrading an Arch Linux system - would be a dream for every server.
- nicoco 4y ago
  Funny, a stable release is the main reason why I choose Debian over anything else. I enjoy that it doesn't change too much. And for the occasional software for which I really want (rarely need) the upstream version, backports, flatpak, docker or compile from sources got me covered. But the beauty of libre software is its diversity. Keep on using what you like ;)
- viccuad 4y ago
  it exists: either Debian Testing (a week or so of proven stability) or Debian Unstable (latest and greatest). If one follows the best practices for them[1] (installing apt-listbugs and apt-listpackages), one gets information of bugs and API-breaking changes before performing the upgrade, and can decide to pin the specific package as needed, and it will automatically get released when the bug is solved. This is thanks to how the Debian packages treat bug reports, and the debian/NEWS file. I can't believe other distributions get it so wrong for so many decades. Plus, these 2 distributions (Testing and Unstable) get all the automated QA usual from Debian: autopkgtest integration tests, piuparts, lintian, reproducible builds, policy conformancy.. It seems it's no coincidence that gLinux from Google, rolling release, and in TFA, is based in Debian. [1]: https://wiki.debian.org/DebianUnstable#What_are_some_best_practices_for_testing.2Fsid_users.3F https://wiki.debian.org/DebianUnstable#What_are_some_best_pr...
  - SkeuomorphicBee 4y ago
    No, it didn't exist (outside Google). Debian testing is not a rolling release because it is not a "release", it lacks internal coherence. As the article mentioned, testing packages are built in unstable and then simply moved to testing individually, with no regard to moving all the build dependency chain together. So you end up with packages in testing that were built using packages that are not in testing yet. Edit: also testing lacks timely security updates, they don't bother to cherry pick security patches for testing and instead just wait until the new secure version make is way thru unstable.

The more important question is where is this magical Google-blessed rolling release Debian available to download? We've all heard rumors of this internal Google distro and what I'm reading is their intent to give back. You can use Microsoft's CBL-Mariner today. Where is Google's?

AdmiralAsshat 4y ago
Why yes, they could share Goobuntu with the world. But they don't have to, and so they won't.
- leoh 4y ago
  It’s really not that interesting
etbusch 4y ago
It seems like they are just using Debian testing with periodic version freezes. It's very likely not all that different from just running testing.
brandon 4y ago
Most of the stuff that makes the distribution different from Debian Testing is tightly coupled with Google's internal infrastructure and generally uninteresting (or unuseful) to users writ large. Just run Debian!
qmarchi 4y ago
If you want to get really technical, there's not much difference between Google's gLinux and Debian testing. Notably, our security, provisioning, and tooling is installed, and that's about it. Nothing that's particularly useful outside of the corp bubble. Disc: Googler, not in CorpEng.
- ryanobjc 4y ago
  Not to mention many of those things don't work outside the rarified environment that is a fully managed VLAN. Regarding "security", remember plenty of security is things like auditing, remote logs, ability of remote admins to control machines, etc, etc. It's ultimately all boring, but essential, large fleet management stuff.
POPOSYS 4y ago
- DiggyJohnson 4y ago
  I have never seen a gender-based criticism of the fragmentation of Linux distros, that’s for sure…

Barrin92 4y ago

personally I'm a pretty big fan of opensuse tumbleweed for a rolling distribution because they ship fully tested snapshots each day rather than individual packages. If I understand the article correctly Google's Sieve is similar to the Opensuse Build Service in that regard. It seems like they went for Debian because of the proximity to Ubuntu but I wonder if they ever considered Suse, the ecosystem is pretty good.

themodelplumber 4y ago
Thanks for your comment. I have been thinking about Tumbleweed or Manjaro to try out a rolling release. Not sure yet which to choose, as they seem quite different in a lot of ways.
compiler-guy 4y ago
The upgrade-in-place plan is made more difficult the more distant from Ubuntu you get. Suse has a great ecosystem, but the transition costs would have been quite a bit higher.
abirch 4y ago
I liked Suse before Novell acquired it. Is it still good?
- Arnavion 4y ago
  Yes. I've been using it for over a decade and it's still great. Like Fedora it's leading the charge in modernization (btrfs, /usr/etc, snapper, Yast, etc) and like Arch it has a big set of up-to-date packages, except they're in the main repo instead of an untrusted AUR.
  - nyanpasu64 4y ago
    You need third party repositories if you want full ffmpeg with h.264 support, and since these are binary (IIRC) repos, each third-party package is often either compatible with the latest or older snapshot, or tumbleweed, but often not both (unlike Arch's AUR which only has one distro to target, though it's a moving target).

lynguist 4y ago

Can I ask the following as this what Google did is literally what I try to achieve on a Raspberry Pi: I have configured unattended-upgrades (the package from apt) on a Raspberry Pi on Raspbian 10, but no security updates seem to ever be released. Is there a gotcha I’m missing resp. is there a better way of configuring an always-on computer and keep it secure?

pertbert 4y ago
The article mentions: > We build packages in package groups, to take into account separate packages that need to be upgraded together. Once the whole group has been built, we run a virtualized test suite to make sure none of our core components and developer workflows are broken. Each group is tested separately with a full system installation, boot and local test suite run on that version of the operating system. > We then proceed to carefully guide this release to the fleet utilizing SRE principles like incremental canarying and monitoring the fleet health.
StillBored 4y ago
Raspbian, or raspberry pi os? The latter is much closer to debian AFAIK, there is a 64-bit version/etc. But if your using it for server tasks, your probably better off actually running Debian. Although if your looking for something that is closer to a rolling release, you could just pick one of the distro's that have a release policy that strikes your fancy WRT how they roll out new packages/etc. Most distro's have rpi specific images, or you just install their generic arm64 images on top of something like the PFTF firmware. Put another way, raspberry pi os, has a fairly dated kernel/etc but works well with all the hardware on the machine for desktop purposes. If your using it as a server, its not a good choice because they frequently don't have useful things turned on (various filesystems, networking options, RAID levels, etc). The core storage/networking/cpu/etc type of things have been working quite well in mainline/etc for a couple years now, so that has trickled down into all these distros.
sillystuff 4y ago
It sounds like you missed a step. You need to create an apt conf file that runs unattended-upgrades (in addition to the configuration you already did in /etc/apt/apt.conf.d/50unattended-upgrades). There are a lot of options (see /usr/lib/apt/apt.systemd.daily for a description of options; the manpages are lacking), but just running: dpkg-reconfigure unattended-upgrades will create a minimal config file, '/etc/apt/apt.conf.d/20auto-upgrades' that may be all you need. But, you might consider adding something like below to cleanup downloaded package files: // Auto cleanup archives APT::Periodic::MaxAge "10"; // Setting minage too is a good idea to prevent race conditions APT::Periodic::MinAge "8"; Or, if you prefer (but, this will delete all archives present when it triggers and not just old ones like above): APT::Periodic::AutocleanInterval "10"; (Note: I'm assuming Raspbian is the same as the upstream Debian; I've never used Raspbian)

glouwbug 4y ago

Just use Arch

whateveracct 4y ago
Or NixOS unstable hehe
- zamalek 4y ago
  The nice thing with flake-based NixOS is that it's trivial to cherry-pick unstable onto a stable base. I do a bunch of that in my nixconfigs: https://gitlab.com/jcdickinson/nix https://gitlab.com/jcdickinson/nix
  - whateveracct 4y ago
    Thanks for sharing. I'm new to flakes, so seeing that idiom of having a separate nixpkgs-unstable arg propagated around set off a lightbulb!
Pr0ject217 4y ago
EndeavourOS here (arch-based). Love it.
- glouwbug 4y ago
  Nothing like a good ol' 9 month late pacman -Syyu
  - Shared404 4y ago
    I dunno, a late xbps-install -Su may give it a run for it's money
dinvlad 4y ago
I wanted to give it a try, but SecureBoot is refusing to be set up correctly, despite following the manual '( (I suspect the problem is with NVidia drivers..)

dxneezal 4y ago

I tried to search more regarding Rodete then found this[1] > … was to get off of Ubuntu's more classic LTS release cycle, and move to the freely-rolling waters of Debian Unstable. An underpublicized but very real reason was that the Google Linux team had a number of Debian Developers on it, who reasonably wanted to be closer to their primary love and affiliation :blink :blink 1: https://www.reddit.com/r/debian/comments/j4liv4/whats_going_on_with_glinux/ https://www.reddit.com/r/debian/comments/j4liv4/whats_going_...

dredmorbius 4y ago

Alt link: https://archive.ph/42l8J https://archive.ph/42l8J

sytse 4y ago

Cool article. My two takeaways: 1) If your deployment process hurts do it it more often (small releases frequently are easier than few big bang releases. 2) Don't commit to being x days behind upstream, if there is a big workload on your team (upgrading, vacations, etc.) you have the flexibility to delay upgrades and reduce the stream of incoming issues.

baby 4y ago
This is true for your codebase and your dependencies as well.
- erik_seaberg 4y ago
  This, remaining on a known-good release is a form of technical debt to use judiciously and pay down when more convenient. Beta testing everything immediately can be a favor to the community, but it’s probably not your main job.
  - yazaddaruvala 4y ago
    Shouldn’t need to be anyones job. Automate: Daily chron: - Branch the repo, auto-update one dependency (ideally a smarter way to batch up groups) - Run CI - Auto-merge commit if CI passes, else discard commit. - Loop to create a new branch for the next dependency waiting to be auto-updated. Otherwise, not having the right testing or CI is technical debt like the grand parent commenter suggests.
  - jillesvangurp 4y ago
    Actually, it's the opposite in my experience. Beta testing is typically done when most software is released. Basically, at that point your technical debt is all the known & fixed issues that you still have, the features you don't have access to and can't benefit from, the performance issues you still have, the newly deprecated APIs that you are still using, etc. Opting into all that without good reason is a bad idea. If you are afraid things will break, the best way to find out is to just try it. Worst case you have to roll back a change. But at least then you know and you can plan a fix. IMHO anything that isn't on a current version should have a documented reason why that is or be fixed ASAP. I rarely see this on projects. Usually that means nobody cared enough to try which is just a form of negligence. On most of my projects I update dependencies whenever I can. On my older projects, I don't even touch any source code until after I update dependencies whenever I do maintenance on them. Typically either nothing breaks, I need to do some minor fixes, or I need to temporarily roll back some dependency to plan some bigger change. The thing is, if a new version is going to require any kind of non trivial work, I want to know about this as early as possible; especially if it is a lot of work. If you wait a year, you are looking at a lot of unknown work, which is basically technical debt you did not even know you had. I don't like that kind of uncertainty. Mostly staying on top of things minimizes the work you actually have to do. And you get to benefit from all the fixes early. A lot of projects I join are hopelessly outdated. It's usually the first thing I fix. It's rare to find documented reasons why a particular thing can't or shouldn't be updated. If it's not documented, I'm just going to go ahead and fix it. In case it doesn't work, I'll document it.

TheRealDunkirk 4y ago

I always thought creating your own rolling distro with Gentoo was precisely the best case use of portage, but I’ve never seen anyone do it. I must be missing something.

tyingq 4y ago
Guessing because a fair amount of stuff is tested, packaged, or unintentionally tied to how either Debian, Ubuntu or RedHat does things (or places things, etc). So anything outside that universe doesn't get popular.
euank 4y ago
Coincidentally, Google has sorta done that too! ChromiumOS/ChromeOS is built using portage, and is certainly rolling release. It's not really a traditional linux distro, but it's something.
- antoncohen 4y ago
  Container-Optimized OS is based on Chromium OS and is more of a traditional Linux, in the sense that is runs on servers. https://cloud.google.com/container-optimized-os/docs https://cloud.google.com/container-optimized-os/docs
  - eliaspro 4y ago
    ...and so was CoreOS before the equally named company was acquired by RedHat.
    - kaeso 4y ago
      In a funny twist of events, the author of the grandparent comment above yours was indeed building that OS (Container Linux) at CoreOS :)

YPPH 4y ago

Out of curiosity, why would a Googler elect to use a Linux based workstation as opposed to a macOS workstation? Are they testing Linux binaries? Developing kernel extensions? Linux build tools? macOS offers such a superior desktop experience I can't imagine why else you'd ask an employer offering you a macOS machine to substitute it with a Linux one.

vinkelhake 4y ago
The average Googler doesn't get to pick the OS that their workstation runs. It's going to be Linux. The typical work setup is a browser, some kind of editor and terminals - which is something that works well in Linux.
- qmarchi 4y ago
  It depends on the team, and the individual. Generally, a Googler does actually get to pick their OS, it's part of the onboarding process that every Noogler goes through, and you're given 30 days from your start to swap around and test the other platforms (gLinux, macOS, gWindows, or Chrome OS).
  - vinkelhake 4y ago
    I was under the impression that unless you have an actual business need that calls for something other than gLinux, then it's going to be gLinux. Like a Chromium dev who primarily works on the Windows side of Chrome obviously has a need for it. I'll fully admit that my onboarding experience is quite out of date since I've been there for a decade. When I joined I didn't really have a choice for workstation since I only worked on server side stuff. Sounds like things have changed.
    - esprehn 4y ago
      No they're the same. Pretty much all google3 work is done on Linux, but folks usually get 2 machines: a laptop you pick the OS and the workstation (or cloudtop) which is Linux unless there's a special need like chrome eng. Lots of folks do stuff like use Chrome OS + cider + cloudtop though.

colordrops 4y ago

I'm surprised no one has mentioned NixOS. Easy rolling forward or back, with config, system snapshots, and cached binaries. Can roll individual packages or the whole system very easily.

pbamotra 4y ago
From the article, > Rolling releases with Linux distributions today are getting more common (Arch Linux, NixOS).
- uncletaco 4y ago
  NixOS users are socialized to not bother reading
  - TomK32 4y ago
    Can't roll back those HN comments though...
    - djbusby 4y ago
      Can edit and/or delete within some time window.
  - colordrops 4y ago
    I'm talking about the comments.
  - SuperSandro2000 4y ago
    Btw I am using NixOS

pmontra 4y ago

Does anybody know which laptops is Google using to run their Linux distro on?

smueller1234 4y ago
Carbon X1's are common. It's not exclusively those though.

akdor1154 4y ago

Jesus.. here i am in my BigCorp being disallowed to dev with Linux because too much mandated corporate crapware doesn't work. I dream of the day my employer provides an in-house distro for me.

HyperSane 4y ago
WSL2 works really well.
- p_l 4y ago
  As part of the crapware image process, WSL (1&2) is often blocked
- happymellon 4y ago
  Unlikely to work in any corporate environment that I've witnessed. This is why so many devs I know moved to Mac's. Not because Mac's are necessary that great, who cares about battery life when it just sits on a desk, but because it's the closest you can get to Linux in a corporate environment. I would take native Linux on an ThinkPad as first option, but denied that I would go for Mac over Windows. I don't find WSL fixed all the other Windows problems such as lack of support for multiple desktops. They exist but the implementation is terrible.
  - surban 4y ago
    Windows 11 natively supports multiple desktops.
    - happymellon 4y ago
      Is it better than 7 or 10? As those "supported" multiple desktops, but was implemented in such a bad way. Gnome and KDE are so much smoother when it comes to these workflows.
  - SuperSandro2000 4y ago
    Mac's are closer to BSD than Linux and suck for Linux oriented development. Brew is more ductape than a solution and you basically need to replace all GNU tools since they are horrible out of date.

deleted 4y ago

[deleted]

walrus01 4y ago

regular debian testing, unless you are doing something really esoteric, is really quite stable for desktop use. such as you might get if you installed the basic barebones debian CLI setup totaling about 1GB disk space occupied and then used apt to install xorg and xfce4 or any other window manager. debian-stable is extremely conservative in packages and stuff is often several years behind.

fulafel 4y ago
What about security updates? They might come slowly to unstable (since security team doesn't concern themselves with unstable) and might be held up arbitrarily long before trickling down to testing, no?
deleted 4y ago
[deleted]

wiseowise 4y ago

Why not ChromeOS?

denysvitali 4y ago
They use ChromeOS as well, this article focuses on those people who want to use a mainstream Linux Distro at Google.
- mjg59 4y ago
  Pretty much everyone has a gLinux workstation or logs into a gLinux VM - they were, traditionally, the only devices that had access to the main source repository, so it's not so much about whether people want to or not.
  - Morgawr 4y ago
    Googler and ChromeOS engineer here. Opinions are my own, etc etc. This has changed/is changing. It heavily depends on your team (especially if you're not in engineering, I don't think most non-engineers would have a gLinux machine or a workstation out of the box). We've developed tools for crostini that would let people access a gLinux environment, I don't know how much I can talk about it so I won't go into details (although it's nothing super secret either), but the bottom line is that more and more engineers these days are able to do 100% of their job on a ChromeOS machine, including dealing with google3 stuff.
    - saagarjha 4y ago
      I mean, you can always use the web-based Cider to do your work. It's not great, but it gets the job done.
alpb 4y ago
Likely because these are not end-user machines but developer machines. Most Google engineers don't checkout/write/build/test/run code on their machines (they either do these tasks through a browser or SSH to another machine). I believe gLinux can be installed on trusted machines that you connect over SSH as well as client laptops (where you can't checkout code etc). That said many engineers/non-engineers at Google use ChromeOS on their client laptops (same with macOS).

raffraffraff 4y ago

Glad to see that they're planning on contributing more to the upstream. But considering the size and budget of Google, and the amount of free work they're getting from the Debian community, it would be utterly irresponsible not to. Not blaming the team that's doing this good work though - I'm sure they're working hard enough as it is.

ryanobjc 4y ago
I think there's a big misunderstanding as to what exactly gLinux _IS_. Nearly every large company I've worked at worked hard to standardize the dev environment. Ensuring the right set of dev tools (and more) is available on every machine. It's not as simple as 'apt-get install build-essential' and you're done. For example many large companies use perforce, which also has a licensing requirement. Google in particular has a large set of custom binaries, remote file systems, unified login (think krb but maybe not exactly that), unified management, etc, etc. That is what gLinux is. This isn't something you're going to want or need. In fact it doesn't even run outside of the Google hard wired VLANs really (for setup). There are a LOT of net benefits for much of google to being on gLinux. It means, for example, that nearly every product works on Linux - google docs, spreadsheet, meet, etc. That's a huge win. Not only that, they are on par with Windows or Mac support (thanks browser as a platform).
- Zhyl 4y ago
  While this is true, I think a consumer gLinux (i.e. a Debian spin with support from Google) would be more appealing to Vendors than Ubuntu and similar just because of brand recognition. I think Google had the opportunity to accidentally displace some of the Windows monopoly, not by doing a land grab a la ChromeOS or Android, but instead by just releasing a version of something they use themselves which would be appealing to like minded developers and companies that would follow Google's 'stamp of approval'.
baby 4y ago
Open source = no strings attached. I don’t get the shaming of big users of open source. (And if we really want to get there: chrome, android, angular, etc.)

pabs3 4y ago

I'm using Debian testing for years with unattended-upgrades upgrading everything 4 times daily (once for each Debian repository update). Not at Google. AMA!

encryptluks2 4y ago
Switch to Arch for God's sake
yasinaydin 4y ago
Have you ever used/considered using another rolling release? If yes, how would you compare them to Debian testing? Also, are there any parts of Debian testing that is not rolling (i.e. OS version number or some package versions like kernel or Node.js)?
- pabs3 4y ago
  I've always used Debian since I discovered it via Knoppix (which was my first Linux distro, Cygwin was my first Unix distro), I initially used Debian unstable but downgraded to testing some years later as upgrading testing often is smoother than unstable. I've never considered even trying any other distro. Everything is rolling, except unmaintained software, that just gets removed from the distro when other faster rolling things change incompatibly.
RealStickman_ 4y ago
Why 4 times daily and do you do that while you're working? How do you deal with packages being broken until the next reboot if you do updates during the day? I don't even update my Arch boxes that regularly and always do a reboot afterwards.
- pabs3 4y ago
  The Debian archive is updated 4 times daily, so I update my system after each Debian update. I do that no matter whether I am working or not. Packages almost always aren't broken in Debian testing but if they were, a reboot would not fix them anyway. I restart processes instead of rebooting (using the needrestart/needrestart-session tools). I only reboot after Linux kernel image/module updates or microcode updates, because Debian cannot yet live-update those components.

baby 4y ago

Interesting post! I’ve been thinking about trying Linux with a rolling distrib. They mention NixOS, do people here have an opinion on that one?

SuperSandro2000 4y ago
NixOS is programmed in a functional programming language and aims to be fully pure and reproducible. Using it since over a year now and my entire hackspace runs on it. Can highly recommend it.
Jedd 4y ago
Why not Debian testing branch, as per TFA?
funcDropShadow 4y ago
I am not sure I would call NixOs a rolling release distribution. It has releases every 6 month like Ubuntu, but it does not have LTS releases. And due to the relatively small community only the latest release is supported. I am doing software development on a desktop workstation, that followed all releases since the start of 2018. And overall it has been rock solid. If an update, either a new release or a config change of mine, breaks something important, I reboot with the previous setup, and I can continue to work until I have time to investigate what is going on. I've used that perhaps 5 times since 2018. But when I used it, it was absolutely critical. But expect a steep learning curve.
- kevincox 4y ago
  nixos-unstable is a rolling release. I use it daily and on all of my servers.
kevincox 4y ago
I love NixOS. I would 100% recommend it for servers and probably 70% recommend it for desktop usage. Feel free to ask me anything but instead of repeating past points I'll just link some past writings: - https://kevincox.ca/2015/12/13/nixos-managed-system/ https://kevincox.ca/2015/12/13/nixos-managed-system/ - https://kevincox.ca/2020/09/06/switching-to-desktop-nixos/ https://kevincox.ca/2020/09/06/switching-to-desktop-nixos/ - https://kevincox.ca/2021/05/06/workstation-install-with-nixos/ https://kevincox.ca/2021/05/06/workstation-install-with-nixo...

Wronnay 4y ago

Rolling Releases are basically my main reason why I choose Arch Linux over Debian. It would be nice, if Google and Debian could team up together to bring a Debian Version with rolling releases. Imagine the stability of Debian packages together with the easiness of upgrading an Arch Linux system - would be a dream for every server.

nicoco 4y ago
Funny, a stable release is the main reason why I choose Debian over anything else. I enjoy that it doesn't change too much. And for the occasional software for which I really want (rarely need) the upstream version, backports, flatpak, docker or compile from sources got me covered. But the beauty of libre software is its diversity. Keep on using what you like ;)
viccuad 4y ago
it exists: either Debian Testing (a week or so of proven stability) or Debian Unstable (latest and greatest). If one follows the best practices for them[1] (installing apt-listbugs and apt-listpackages), one gets information of bugs and API-breaking changes before performing the upgrade, and can decide to pin the specific package as needed, and it will automatically get released when the bug is solved. This is thanks to how the Debian packages treat bug reports, and the debian/NEWS file. I can't believe other distributions get it so wrong for so many decades. Plus, these 2 distributions (Testing and Unstable) get all the automated QA usual from Debian: autopkgtest integration tests, piuparts, lintian, reproducible builds, policy conformancy.. It seems it's no coincidence that gLinux from Google, rolling release, and in TFA, is based in Debian. [1]: https://wiki.debian.org/DebianUnstable#What_are_some_best_practices_for_testing.2Fsid_users.3F https://wiki.debian.org/DebianUnstable#What_are_some_best_pr...
- SkeuomorphicBee 4y ago
  No, it didn't exist (outside Google). Debian testing is not a rolling release because it is not a "release", it lacks internal coherence. As the article mentioned, testing packages are built in unstable and then simply moved to testing individually, with no regard to moving all the build dependency chain together. So you end up with packages in testing that were built using packages that are not in testing yet. Edit: also testing lacks timely security updates, they don't bother to cherry pick security patches for testing and instead just wait until the new secure version make is way thru unstable.