10 ms·

Ubuntu 22.04 LTS servers and phased apt updates

256 points by goranmoomin 4y ago

squiffsquiff 4y ago
IIRC this is similar to the experience with Windows where people have been struggling for years to get consistent patch states
- sokoloff 4y ago
  We struggled a bit and largely succeeded in consistency by hand—picking the patches and using automation to drive them. What we couldn’t do was get to both consistent and really timely patching states but this was a decade ago where that was arguably slightly less important.
kramerger 4y ago
TL;DR: a decade ago Ubuntu wanted to update a few computers first, then more computers if no issues were detected. Due to missing tooling, this didn't hit mainstream until now. A practical problem with these phased updates is that you may end up with different versions of some software on your fleet of (until now) identical servers and you have no way of avoiding it.
- danw1979 4y ago
  > then more computers if no issues were detected. .. a job they should have left to the sysadmins in charge of those machines receiving said updates.
  - charcircuit 4y ago
    Why do you think pushing out broken updates to people and causing them to do extra work is a good thing? It's something that should happen as little as possible. If the OS developers have the ability to save thousands of man hours of their users they should do so.
  - SahAssar 4y ago
    Ubuntu does a lot of things in both server and desktop that might only make sense on desktop (if even there).
- ekianjo 4y ago
  You do have a way of avoiding it as mentioned in the link by setting an identical machine id string
  - AshamedCaptain 4y ago
    And then after reusing /etc/machine-id everywhere you spend the day after that debugging miscellaneous systemd issues.
    - drothlis 4y ago
      In the article they don't change /etc/machine-id, but APT::Machine-ID in apt.conf.
- brnt 4y ago
  Who runs a fleet without their own apt cache server?
  - Symbiote 4y ago
    I do, since there's an official apt mirror in the same datacentre.
    - brnt 4y ago
      That should isolate you from this issue, shouldn't it?
  - soneil 4y ago
    I don't believe mirrors and caches affect this behaviour. How this works is that the package index contains a stanza Phased-Update-Percentage: giving a number between 0 and 100. Then your client picks a number in the same range, and accepts an update if the repository's value is higher than the client's value. Whether the fixed value comes from your mirror or Ubuntu's, makes no odds - You still have a fixed value on the server and a variable value on the client, so the outcome is still variable. Caching the index, or mirroring it without regenerating it, still leaves this decision to the end client. Something else I found interesting looking into this mechanism - the seed for the rng is sourcePackage-version-machineID. So you won't "that one machine that always updates last" or "that one machine that charges head-first into phased updates", it should be randomly distributed for each version of each package.
- philjohn 4y ago
  There is a way to avoid it, disable phased updates as detailed here: https://news.ycombinator.com/item?id=34388575 https://news.ycombinator.com/item?id=34388575
itrollpussies 4y ago
[dead]
Jedd 4y ago
First I've heard of this, and as a long-time Debian user it initially sounded like yet another oddity peculiar to Ubuntu. It sounds like this problem shouldn't / wouldn't affect most 'business' servers iff you were running the LTS (stable) branch, as security updates aren't phased (as per TFA's comment). LTS Ubuntu, AIUI, is similar to Debian stable -- it only receives security patches, not new (feature) versions of any packages. Reviewing apt & apt-get man pages here (I'm on 2.5.4) I see no references to 'phase'. Reviewing Debian's changelog.gz I see the first phase reference in 2.5.1 (2022-07), and a subsequent reference in 2.5.3 (2022-09) - but nothing about 'phased updates' on the Debian wiki, and per Ubuntu's discourse[0] phased updates were first introduced to apt in version 2.1.16 - so I'm guessing it took a while for those to be fed upstream (to Debian - assuming they are still considered upstream for - dare I say it, the canonical owners of - apt/apt-get). There's an askubuntu [1] post describing the what & why behind this change, and it sells the reader on the 'improved stability' claim, while downplaying TFA's concerns (potential random inconsistency), while acknowledging poor defaults, tooling, and documentation. It does feel a bit of an odd solution to me - Debian provides a testing branch which seems to serve this function - some relatively small subset of users will use testing and find your bugs for you. If you seek safety & predictability then you stick with the stable / LTS branch. In contrast, randomly selecting some subset of your users - with Ubuntu's typical opt-out default - to randomly get an early / deferred updating of some random subset of your installed packages just seems ... well, I can see why TFA was frustrated. [0] https://discourse.ubuntu.com/t/phased-updates-in-apt-in-21-04/20345 https://discourse.ubuntu.com/t/phased-updates-in-apt-in-21-0... [1] https://askubuntu.com/questions/1431940/what-are-phased-updates-and-why-does-ubuntu-use-them https://askubuntu.com/questions/1431940/what-are-phased-upda...
- usr1106 4y ago
  Will Debian upstrem even accept the feature? From what the blog post says the implementation is user-hostile: No logging what is going on and limited choice, no way to run a canary.
  - Jedd 4y ago
    I've updated my earlier comment to clarify that I was checking the changelog for apt in my Debian system. On review it doesn't make much sense - searching for 'phase' there's an 'Add support for phased updates' in 2.5.3, but a few months earlier, in 2.5.1, the comment is: "(Temporarily) Rewrite phased updates using a keep-back approach." I thought I'd done a case-insensitive search before, but obviously hadn't, as there's one earlier reference in 2021-01 (for v2.1.16), which kind of aligns with the askubuntu's historical mention of 2.1.16 on the timeline, but the description certainly doesn't sound like the base feature (phased updates) introduction. "Add support for Phased-Update-Percentage, previously used only by update-manager."
usr1106 4y ago
Fedora CoreOS has phased updates, too. But being atomic updates of the whole distribution there is no risk of inconsistencies between packages. Either your machine has the new version or not. And the user can choose a priority, do they want updates early or late, so you can run a canary. Not a convincing feature from Ubuntu with such poor tooling, limited user choice and no logging.
- stingraycharles 4y ago
  Maybe, but there’s still some state in eg config files that persists over upgrades. If there are incompatibilities there, you’ll still have issues, so it’s not a silver bullet.
- AshamedCaptain 4y ago
  It's not possible to get inconsistencies here either unless you force it, which is kind of the intention.
  - usr1106 4y ago
    Given that no bugs exist, it's not possible to get inconsistencies by installing packages in random order because they declare their dependencies. If no bugs existed we would not need updates at all...
    - p4l4g4 4y ago
      That's why we only allow RFC9225 compliant packages on our systems!
    - AshamedCaptain 4y ago
      And assuming bugs exist, you can get inconsistencies everywhere, so I fail to see the point of this message.
cosmin800 4y ago
Ubuntu is known to break things, quite often, last one that comes to my mind was the sudo behaviour: On Wed, 2019-05-15 at 02:42:56 +0930, Dan Streetman wrote: > in Ubuntu, sudo retains the calling user's $HOME > > this is different from upstream sudo as well as all other UNIXes and > even the sudo documentation we provide. Should we remove our custom > patch that adds this behavior? Ubuntu is diverging.
- tankenmate 4y ago
  Wow, that is a surprise. Personally I've never hit it because I use "sudo su -".
  - stefantalpalaru 4y ago
    > I use "sudo su -" You can simplify that to "sudo -i".
  - cuillevel3 4y ago
    Isn't that identical to 'sudo -i'?
    - RealStickman_ 4y ago
      It gives you a pretty similar result in the end. From my understanding, with 'sudo -i', you're still using sudo itself to run commands as root (or any other specified user). 'sudo su -' instead executes the 'su -' command, giving you a root shell, as a superuser with 'sudo'. If you left the 'sudo' out, you'd have to type the root password.
      - tremon 4y ago
        sudo -i and sudo -s also give you a root shell. "sudo su" is a tautology that's unnecessary is almost all cases.
        yrro 4y ago
paol 4y ago
I noticed this too starting with 21.10 (I think?), and it eventually bothered me so much I went looking for answers and found out about this. This was an unusually badly communicated change. I'm fairly attentive to this type of stuff and was caught by surprise. We're about to start rolling out 22.04 at work. I'll make sure to disable this on servers, non-determinism is not a desirable property for system updates.
3np 4y ago
At the scale of OP with their particular needs (and wishes for canary rollouts), shouldn't it already be about time to host your own internal mirrors regardless? Then you get full control.
deleted 4y ago
[deleted]
nyx 4y ago
Ahh, /etc/machine-id, we meet again... Last time I met you, you were causing my cloned VM to get the same DHCP address as its original and blow up my VM network, despite libvirt's virt-clone utility properly randomizing vNIC MAC addresses, because netplan defaults to using you as the DHCP client identifier. If only we could meet under more pleasant circumstances.
- cosmin800 4y ago
  this is not an ubuntu only issue, happens on debian too
- dark-star 4y ago
  Can't you just get rid of that file entirely? I would assume that there is a (sane) fallback for DHCP in case this file is missing? What other uses besides the phased updates and DHCP does that file solve?
  - tremon 4y ago
    Google Chrome reads it, probably for tracking purposes.
- usr1106 4y ago
  As another comment pointed out elsewhere you are not supposed to touch /etc/machine-id to control apt. It has its own Apt::Machine-ID you can change without breaking anything else.
  - yjftsjthsd-h 4y ago
    That's almost worse in my opinion. If you're going to have unique machine ids, is it too much to ask that there's only one to deal with?
    - Denvercoder9 4y ago
      In the default configuration apt uses /etc/machine-id, it just has its own override (which is reasonable, imo).
AnonCoward42 4y ago
Is that the reason Ubuntu installs half updates and breaking the system the last few months? Three times now the kernel was updated but not the nVidia modules, leaving the user with a system without the actual graphics drivers. And once the wrong kernel was installed for an unknown reason (oem package installed, but I never selected it). After a manual update it worked once again. Between the update and the fix there might be a few days in between tho as the updates were automatically installed, but the effect/issue can only be seen after a restart.
- crakenzak 4y ago
  And once; * Sorry had to correct, for my own understanding =)
rurban 4y ago
Extremely bad advice given on this page. Don"t mess with the machine-id! Rather turn off phased updates. Set this in some of your apt.conf.d: Update-Manager::Always-Include-Phased-Updates; APT::Get::Always-Include-Phased-Updates;
- PBondurant 4y ago
  Note that this code actually turns phase-updates ON. To turn them OFF: Update-Manager::Never-Include-Phased-Updates; APT::Get::Never-Include-Phased-Updates: True;
  - proactivesvcs 4y ago
    Perhaps the advice is bad because this doesn't appear to be mentioned in the documentation.
  - soneil 4y ago
    As I understand it, either 'always' or 'never' actually solve OP's problem. (edit: was both/and - both sounds like a bad idea.) "always include" effectively puts you in phase 0 - if an update is being phased, you're an eager beaver. "never include" effectively puts you in phase 100 - if an update is being phased, you'll wait until the phasing is complete. AIUI OP's problem isn't that he wants these updates on day 0, it's that he wants his environments to be consistent with each other, which either of these options would provide.
- lima 4y ago
  Adding to that, the machine ID is a global identifier and changing it could lead to all sorts of unpredictable consequences elsewhere.
  - deleted 4y ago
    [deleted]
  - pixl97 4y ago
    Hmm, time to fetch some random characters on each boot for this file and see what starts going on.
throwawaaarrgh 4y ago
The wonderful world of corporate OSS. Your needs come second to the company. Remember, kids: if you're not paying for it, you're the product.
Klasiaster 4y ago
> Now that I've looked at all of this and read about APT::Machine-ID, we'll probably set it to a single value across all of our fleet This is not a good idea, the machine ID is supposed to be unique and shouldn't change over the lifetime - a handful of software relies on this property and it's the best identifier for an installation if you can't rely on hostnames.
- _cenw 4y ago
  APT::Machine-ID overrides it only for APT, you can still have a unique machine-id in /etc
wongarsu 4y ago
The article repeats that they would like to set up a canary server but can't. But they also link to an askubuntu answer which mentions these two settings: Update-Manager::Always-Include-Phased-Updates; APT::Get::Always-Include-Phased-Updates: True; Update-Manager::Never-Include-Phased-Updates; APT::Get::Never-Include-Phased-Updates: True; Surely the best setup would be to set some canaries to always include phased updates, and the rest of the fleet to never include them (so you get them once they are at 100% rollout and no longer phased)?
- imp0cat 4y ago
  Absolutely. Also, no need to fix machine id across the entire cluster.
- dinosor 4y ago
  True. I wondered the same for the canaries. But they did mention why they didn't want to go with the 100% rollout option. > We could set some very important machines to only get updates when packages reach 100% and stop being phased updates, but Ubuntu has a good record of not blowing things up with eg OpenSSH updates.
kmac_ 4y ago
Absolutely not related with the issue: apt started to print some Reddit reference on the screen. I know, not everything has to be serious, but come on, when I deploy an image to a customer I would prefer not to see some random messages.
- jeroenhd 4y ago
  I don't mind the message itself too much, but I think it was still in particular bad taste because the message was introduced after they started spamming ads into your install logs and people got upset. "Haha here's how you disable the shitty ads ;)" isn't something I expected, even from Canonical.
ndsipa_pomu 4y ago
So, that's what happening with the held package updates. I was seeing this on a few boxes, but hadn't looked into why it was happening as it hasn't caused any issues. I understand why there's people here who don't like that behaviour, but I like the idea of non-security phased updates, so I'll be leaving machines with the default behaviour until I encounter a problem caused by it.
secondcoming 4y ago
I'm not sure what's going on here. Does it only apply to `apt upgrade` or also to `apt update`? Is it possible that my local dev environment uses different packages to our CI build system?
markstos 4y ago
It sounds like the issues are avoided if you only apply the security updates.
ilyt 4y ago
I swear, Ubuntu is at that point just taking what works (Debian) and decides to break shit at random
nequo 4y ago
You can also check if a particular package, call it x, is held back by phased updates by running apt-cache policy x
MR4D 4y ago
Early on in the days of OS X (perhaps the first 10 years or so), Apple made a good habit of focusing on one particular subsystem for massive changes, and then tended to only make smaller scale changes across the rest of the OS. I wonder sometimes if that would be too hard to replicate as a Linux OS vender. Seems like the approach worked well and it could again.
- smoldesu 4y ago
  That's kinda how Linux operates, just without a project manager. A good example is the audio subsystem Linux had 2 or 3 years ago - pretty bad, dropped Bluetooth connections all the time and broke on newer systems. So, the community came together and wrote PipeWire to fix these problems. The end-result is uncharacteristically impressive, in my opinion. Still though, the last thing I want the Linux community to do is drop everything to focus on $SOMETHING. The people using Linux are a diverse audience with many different use-cases, and naturally not all of them want another MacOS to babysit.
  - MR4D 4y ago
    My point is more that distributions don’t have to update everything in a release - they can focus their energy on certain new things rather than everything.
CoastalCoder 4y ago
Wouldn't it make more sense for Canonical to either... (a) add several more release channels? E.g., latest-phase-1, latest-phase-2, ..., latest-phase-n, or (b) add highly granular, timestamped releases? E.g., 2023-01-01.1, 2023-01-02.1, 2023-01-02.2 (if needed), etc., or (c) publish explicit rollout-group channels, and let user pick their group if they want? E.g., stable-stage{1...k}-group{1...j}, and also offer an optional technique for admins to participate in a lottery for which group they draw from? Then individual computer owners could decide policy for themselves. I really don't understand how Canonical's views on the proper role of a Linux distro provider could have diverged so far from my own. I think they're the ones who have changed, but I'm not sure.
exabrial 4y ago
I was literally wondering what the hell was going on... applied updates to some servers then came back and saw there were some remaining!
baggy_trough 4y ago
It is highly annoying that one can have beta servers with out of sync updates compared to production servers.
Animats 4y ago
The messages are terrible. Software Update tells you that something went wrong with the update, but doesn't tell you what. Synaptic Package Manager says 0 packages are broken. As someone commented in the bug report, "I have seen many people on IRC very* upset after wasting a lot of time trying to install updates that apt will not let them install. Fixing this is critical to our reputation."* The other big Ubuntu update hassle is a constant string of notifications demanding that you exit applications so they can be updated. However, you have to keep them closed long enough for an updating cycle to notice. And then there's the notification that you need to close the Snap daemon so it can update. The user doesn't start the Snap daemon; it starts at startup and has no desktop presence. Lame. This may have been fixed; I haven't seen that recently.
warystarship 4y ago
I'm running ubuntu 22.04 as my daily driver and managed to get held-back updates to run using "sudo aptitude safe-upgrade". Right or wrong move?
voakbasda 4y ago
I wonder whether this “feature” is a ploy to push more people to their fleet management offering. My own systems are now all out of sync with each other, because they are all in different phases of deployment. After reading some of the replies here, I think that I ought to disable this feature on all of my systems, but I worry that the whole situation foreshadows the possibility that the Ubuntu maintainers want to play more fast and loose with the “stable” release feeds. You know, move fast and break things.

256 points by goranmoomin 4y ago

IIRC this is similar to the experience with Windows where people have been struggling for years to get consistent patch states

sokoloff 4y ago
We struggled a bit and largely succeeded in consistency by hand—picking the patches and using automation to drive them. What we couldn’t do was get to both consistent and really timely patching states but this was a decade ago where that was arguably slightly less important.

kramerger 4y ago

TL;DR: a decade ago Ubuntu wanted to update a few computers first, then more computers if no issues were detected. Due to missing tooling, this didn't hit mainstream until now. A practical problem with these phased updates is that you may end up with different versions of some software on your fleet of (until now) identical servers and you have no way of avoiding it.

danw1979 4y ago
> then more computers if no issues were detected. .. a job they should have left to the sysadmins in charge of those machines receiving said updates.
- charcircuit 4y ago
  Why do you think pushing out broken updates to people and causing them to do extra work is a good thing? It's something that should happen as little as possible. If the OS developers have the ability to save thousands of man hours of their users they should do so.
- SahAssar 4y ago
  Ubuntu does a lot of things in both server and desktop that might only make sense on desktop (if even there).
ekianjo 4y ago
You do have a way of avoiding it as mentioned in the link by setting an identical machine id string
- AshamedCaptain 4y ago
  And then after reusing /etc/machine-id everywhere you spend the day after that debugging miscellaneous systemd issues.
  - drothlis 4y ago
    In the article they don't change /etc/machine-id, but APT::Machine-ID in apt.conf.
brnt 4y ago
Who runs a fleet without their own apt cache server?
- Symbiote 4y ago
  I do, since there's an official apt mirror in the same datacentre.
  - brnt 4y ago
    That should isolate you from this issue, shouldn't it?
- soneil 4y ago
  I don't believe mirrors and caches affect this behaviour. How this works is that the package index contains a stanza Phased-Update-Percentage: giving a number between 0 and 100. Then your client picks a number in the same range, and accepts an update if the repository's value is higher than the client's value. Whether the fixed value comes from your mirror or Ubuntu's, makes no odds - You still have a fixed value on the server and a variable value on the client, so the outcome is still variable. Caching the index, or mirroring it without regenerating it, still leaves this decision to the end client. Something else I found interesting looking into this mechanism - the seed for the rng is sourcePackage-version-machineID. So you won't "that one machine that always updates last" or "that one machine that charges head-first into phased updates", it should be randomly distributed for each version of each package.
philjohn 4y ago
There is a way to avoid it, disable phased updates as detailed here: https://news.ycombinator.com/item?id=34388575 https://news.ycombinator.com/item?id=34388575

itrollpussies 4y ago

[dead]

Jedd 4y ago

First I've heard of this, and as a long-time Debian user it initially sounded like yet another oddity peculiar to Ubuntu. It sounds like this problem shouldn't / wouldn't affect most 'business' servers iff you were running the LTS (stable) branch, as security updates aren't phased (as per TFA's comment). LTS Ubuntu, AIUI, is similar to Debian stable -- it only receives security patches, not new (feature) versions of any packages. Reviewing apt & apt-get man pages here (I'm on 2.5.4) I see no references to 'phase'. Reviewing Debian's changelog.gz I see the first phase reference in 2.5.1 (2022-07), and a subsequent reference in 2.5.3 (2022-09) - but nothing about 'phased updates' on the Debian wiki, and per Ubuntu's discourse[0] phased updates were first introduced to apt in version 2.1.16 - so I'm guessing it took a while for those to be fed upstream (to Debian - assuming they are still considered upstream for - dare I say it, the canonical owners of - apt/apt-get). There's an askubuntu [1] post describing the what & why behind this change, and it sells the reader on the 'improved stability' claim, while downplaying TFA's concerns (potential random inconsistency), while acknowledging poor defaults, tooling, and documentation. It does feel a bit of an odd solution to me - Debian provides a testing branch which seems to serve this function - some relatively small subset of users will use testing and find your bugs for you. If you seek safety & predictability then you stick with the stable / LTS branch. In contrast, randomly selecting some subset of your users - with Ubuntu's typical opt-out default - to randomly get an early / deferred updating of some random subset of your installed packages just seems ... well, I can see why TFA was frustrated. [0] https://discourse.ubuntu.com/t/phased-updates-in-apt-in-21-04/20345 https://discourse.ubuntu.com/t/phased-updates-in-apt-in-21-0... [1] https://askubuntu.com/questions/1431940/what-are-phased-updates-and-why-does-ubuntu-use-them https://askubuntu.com/questions/1431940/what-are-phased-upda...

usr1106 4y ago
Will Debian upstrem even accept the feature? From what the blog post says the implementation is user-hostile: No logging what is going on and limited choice, no way to run a canary.
- Jedd 4y ago
  I've updated my earlier comment to clarify that I was checking the changelog for apt in my Debian system. On review it doesn't make much sense - searching for 'phase' there's an 'Add support for phased updates' in 2.5.3, but a few months earlier, in 2.5.1, the comment is: "(Temporarily) Rewrite phased updates using a keep-back approach." I thought I'd done a case-insensitive search before, but obviously hadn't, as there's one earlier reference in 2021-01 (for v2.1.16), which kind of aligns with the askubuntu's historical mention of 2.1.16 on the timeline, but the description certainly doesn't sound like the base feature (phased updates) introduction. "Add support for Phased-Update-Percentage, previously used only by update-manager."

usr1106 4y ago

Fedora CoreOS has phased updates, too. But being atomic updates of the whole distribution there is no risk of inconsistencies between packages. Either your machine has the new version or not. And the user can choose a priority, do they want updates early or late, so you can run a canary. Not a convincing feature from Ubuntu with such poor tooling, limited user choice and no logging.

stingraycharles 4y ago
Maybe, but there’s still some state in eg config files that persists over upgrades. If there are incompatibilities there, you’ll still have issues, so it’s not a silver bullet.
AshamedCaptain 4y ago
It's not possible to get inconsistencies here either unless you force it, which is kind of the intention.
- usr1106 4y ago
  Given that no bugs exist, it's not possible to get inconsistencies by installing packages in random order because they declare their dependencies. If no bugs existed we would not need updates at all...
  - p4l4g4 4y ago
    That's why we only allow RFC9225 compliant packages on our systems!
  - AshamedCaptain 4y ago
    And assuming bugs exist, you can get inconsistencies everywhere, so I fail to see the point of this message.

cosmin800 4y ago

Ubuntu is known to break things, quite often, last one that comes to my mind was the sudo behaviour: On Wed, 2019-05-15 at 02:42:56 +0930, Dan Streetman wrote: > in Ubuntu, sudo retains the calling user's $HOME > > this is different from upstream sudo as well as all other UNIXes and > even the sudo documentation we provide. Should we remove our custom > patch that adds this behavior? Ubuntu is diverging.

tankenmate 4y ago
Wow, that is a surprise. Personally I've never hit it because I use "sudo su -".
- stefantalpalaru 4y ago
  > I use "sudo su -" You can simplify that to "sudo -i".
- cuillevel3 4y ago
  Isn't that identical to 'sudo -i'?
  - RealStickman_ 4y ago
    It gives you a pretty similar result in the end. From my understanding, with 'sudo -i', you're still using sudo itself to run commands as root (or any other specified user). 'sudo su -' instead executes the 'su -' command, giving you a root shell, as a superuser with 'sudo'. If you left the 'sudo' out, you'd have to type the root password.
    - tremon 4y ago
      sudo -i and sudo -s also give you a root shell. "sudo su" is a tautology that's unnecessary is almost all cases.
      - yrro 4y ago

paol 4y ago

I noticed this too starting with 21.10 (I think?), and it eventually bothered me so much I went looking for answers and found out about this. This was an unusually badly communicated change. I'm fairly attentive to this type of stuff and was caught by surprise. We're about to start rolling out 22.04 at work. I'll make sure to disable this on servers, non-determinism is not a desirable property for system updates.

3np 4y ago

At the scale of OP with their particular needs (and wishes for canary rollouts), shouldn't it already be about time to host your own internal mirrors regardless? Then you get full control.

deleted 4y ago

[deleted]

nyx 4y ago

Ahh, /etc/machine-id, we meet again... Last time I met you, you were causing my cloned VM to get the same DHCP address as its original and blow up my VM network, despite libvirt's virt-clone utility properly randomizing vNIC MAC addresses, because netplan defaults to using you as the DHCP client identifier. If only we could meet under more pleasant circumstances.

cosmin800 4y ago
this is not an ubuntu only issue, happens on debian too
dark-star 4y ago
Can't you just get rid of that file entirely? I would assume that there is a (sane) fallback for DHCP in case this file is missing? What other uses besides the phased updates and DHCP does that file solve?
- tremon 4y ago
  Google Chrome reads it, probably for tracking purposes.
usr1106 4y ago
As another comment pointed out elsewhere you are not supposed to touch /etc/machine-id to control apt. It has its own Apt::Machine-ID you can change without breaking anything else.
- yjftsjthsd-h 4y ago
  That's almost worse in my opinion. If you're going to have unique machine ids, is it too much to ask that there's only one to deal with?
  - Denvercoder9 4y ago
    In the default configuration apt uses /etc/machine-id, it just has its own override (which is reasonable, imo).

AnonCoward42 4y ago

Is that the reason Ubuntu installs half updates and breaking the system the last few months? Three times now the kernel was updated but not the nVidia modules, leaving the user with a system without the actual graphics drivers. And once the wrong kernel was installed for an unknown reason (oem package installed, but I never selected it). After a manual update it worked once again. Between the update and the fix there might be a few days in between tho as the updates were automatically installed, but the effect/issue can only be seen after a restart.

crakenzak 4y ago
And once; * Sorry had to correct, for my own understanding =)

rurban 4y ago

Extremely bad advice given on this page. Don"t mess with the machine-id! Rather turn off phased updates. Set this in some of your apt.conf.d: Update-Manager::Always-Include-Phased-Updates; APT::Get::Always-Include-Phased-Updates;

PBondurant 4y ago
Note that this code actually turns phase-updates ON. To turn them OFF: Update-Manager::Never-Include-Phased-Updates; APT::Get::Never-Include-Phased-Updates: True;
- proactivesvcs 4y ago
  Perhaps the advice is bad because this doesn't appear to be mentioned in the documentation.
- soneil 4y ago
  As I understand it, either 'always' or 'never' actually solve OP's problem. (edit: was both/and - both sounds like a bad idea.) "always include" effectively puts you in phase 0 - if an update is being phased, you're an eager beaver. "never include" effectively puts you in phase 100 - if an update is being phased, you'll wait until the phasing is complete. AIUI OP's problem isn't that he wants these updates on day 0, it's that he wants his environments to be consistent with each other, which either of these options would provide.
lima 4y ago
Adding to that, the machine ID is a global identifier and changing it could lead to all sorts of unpredictable consequences elsewhere.
- deleted 4y ago
  [deleted]
- pixl97 4y ago
  Hmm, time to fetch some random characters on each boot for this file and see what starts going on.

throwawaaarrgh 4y ago

The wonderful world of corporate OSS. Your needs come second to the company. Remember, kids: if you're not paying for it, you're the product.

Klasiaster 4y ago

> Now that I've looked at all of this and read about APT::Machine-ID, we'll probably set it to a single value across all of our fleet This is not a good idea, the machine ID is supposed to be unique and shouldn't change over the lifetime - a handful of software relies on this property and it's the best identifier for an installation if you can't rely on hostnames.

_cenw 4y ago
APT::Machine-ID overrides it only for APT, you can still have a unique machine-id in /etc

wongarsu 4y ago

The article repeats that they would like to set up a canary server but can't. But they also link to an askubuntu answer which mentions these two settings: Update-Manager::Always-Include-Phased-Updates; APT::Get::Always-Include-Phased-Updates: True; Update-Manager::Never-Include-Phased-Updates; APT::Get::Never-Include-Phased-Updates: True; Surely the best setup would be to set some canaries to always include phased updates, and the rest of the fleet to never include them (so you get them once they are at 100% rollout and no longer phased)?

imp0cat 4y ago
Absolutely. Also, no need to fix machine id across the entire cluster.
dinosor 4y ago
True. I wondered the same for the canaries. But they did mention why they didn't want to go with the 100% rollout option. > We could set some very important machines to only get updates when packages reach 100% and stop being phased updates, but Ubuntu has a good record of not blowing things up with eg OpenSSH updates.

kmac_ 4y ago

Absolutely not related with the issue: apt started to print some Reddit reference on the screen. I know, not everything has to be serious, but come on, when I deploy an image to a customer I would prefer not to see some random messages.

jeroenhd 4y ago
I don't mind the message itself too much, but I think it was still in particular bad taste because the message was introduced after they started spamming ads into your install logs and people got upset. "Haha here's how you disable the shitty ads ;)" isn't something I expected, even from Canonical.

ndsipa_pomu 4y ago

So, that's what happening with the held package updates. I was seeing this on a few boxes, but hadn't looked into why it was happening as it hasn't caused any issues. I understand why there's people here who don't like that behaviour, but I like the idea of non-security phased updates, so I'll be leaving machines with the default behaviour until I encounter a problem caused by it.

secondcoming 4y ago

I'm not sure what's going on here. Does it only apply to `apt upgrade` or also to `apt update`? Is it possible that my local dev environment uses different packages to our CI build system?

markstos 4y ago

It sounds like the issues are avoided if you only apply the security updates.

ilyt 4y ago

I swear, Ubuntu is at that point just taking what works (Debian) and decides to break shit at random

nequo 4y ago

You can also check if a particular package, call it x, is held back by phased updates by running apt-cache policy x

MR4D 4y ago

Early on in the days of OS X (perhaps the first 10 years or so), Apple made a good habit of focusing on one particular subsystem for massive changes, and then tended to only make smaller scale changes across the rest of the OS. I wonder sometimes if that would be too hard to replicate as a Linux OS vender. Seems like the approach worked well and it could again.

smoldesu 4y ago
That's kinda how Linux operates, just without a project manager. A good example is the audio subsystem Linux had 2 or 3 years ago - pretty bad, dropped Bluetooth connections all the time and broke on newer systems. So, the community came together and wrote PipeWire to fix these problems. The end-result is uncharacteristically impressive, in my opinion. Still though, the last thing I want the Linux community to do is drop everything to focus on $SOMETHING. The people using Linux are a diverse audience with many different use-cases, and naturally not all of them want another MacOS to babysit.
- MR4D 4y ago
  My point is more that distributions don’t have to update everything in a release - they can focus their energy on certain new things rather than everything.

CoastalCoder 4y ago

Wouldn't it make more sense for Canonical to either... (a) add several more release channels? E.g., latest-phase-1, latest-phase-2, ..., latest-phase-n, or (b) add highly granular, timestamped releases? E.g., 2023-01-01.1, 2023-01-02.1, 2023-01-02.2 (if needed), etc., or (c) publish explicit rollout-group channels, and let user pick their group if they want? E.g., stable-stage{1...k}-group{1...j}, and also offer an optional technique for admins to participate in a lottery for which group they draw from? Then individual computer owners could decide policy for themselves. I really don't understand how Canonical's views on the proper role of a Linux distro provider could have diverged so far from my own. I think they're the ones who have changed, but I'm not sure.

exabrial 4y ago

I was literally wondering what the hell was going on... applied updates to some servers then came back and saw there were some remaining!

baggy_trough 4y ago

It is highly annoying that one can have beta servers with out of sync updates compared to production servers.

Animats 4y ago

The messages are terrible. Software Update tells you that something went wrong with the update, but doesn't tell you what. Synaptic Package Manager says 0 packages are broken. As someone commented in the bug report, "I have seen many people on IRC very* upset after wasting a lot of time trying to install updates that apt will not let them install. Fixing this is critical to our reputation."* The other big Ubuntu update hassle is a constant string of notifications demanding that you exit applications so they can be updated. However, you have to keep them closed long enough for an updating cycle to notice. And then there's the notification that you need to close the Snap daemon so it can update. The user doesn't start the Snap daemon; it starts at startup and has no desktop presence. Lame. This may have been fixed; I haven't seen that recently.

warystarship 4y ago

I'm running ubuntu 22.04 as my daily driver and managed to get held-back updates to run using "sudo aptitude safe-upgrade". Right or wrong move?

voakbasda 4y ago

I wonder whether this “feature” is a ploy to push more people to their fleet management offering. My own systems are now all out of sync with each other, because they are all in different phases of deployment. After reading some of the replies here, I think that I ought to disable this feature on all of my systems, but I worry that the whole situation foreshadows the possibility that the Ubuntu maintainers want to play more fast and loose with the “stable” release feeds. You know, move fast and break things.