3 ms·

> Anything I deploy needs to have an upgrade plan. Ideally, something that provides a package (either on distro or a repo the package provides), so "apt update"

> Anything I deploy needs to have an upgrade plan. Ideally, something that provides a package (either on distro or a repo the package provides), so "apt update" will resolve it. Docker can be a good way as well, Sentry does a pretty good job at this. I wrote an article called "Never update anything": https://blog.kronis.dev/articles/never-update-anything https://blog.kronis.dev/articles/never-update-anything which in truth argued that while updates are necessary, they're also going to break things... a lot. And there isn't always going to be an upgrade path either (e.g. using AngularJS or Clusterpoint). In my experience, even containers break things surprisingly often: everything from GitLab, Nextcloud, OpenProject to even things breaking in regular server updates, like a Debian update breaking GRUB or another Debian install automatically launching exim4 which prevented my own mail server from working. Perhaps that's because of how we build and package software, because of the fact that we don't separate the runtime from the data enough (e.g. persistent directories) or that we make too many assumptions about the environments... Regardless, I can understand why some don't even update working but insecure software: because of the risk to turn it into secure but not working software.

XCSme 4y ago
I agree, I ran many WordPress sites, UXWizz dashboards or even my own tools written in Node.js, and they never ever broke by themselves in 10+ years. The only time there is an issue if I decide to update/change anything. In general, software that has once worked will always work. If I open my PSP (Playstation Portable) from 2005 it will still work as well as in the first day, all the games work the same and the boot time and interface are faster than most consoles nowadays. Why does it still work? Because it once worked and nothing changed.
dicknuckle 4y ago
Anecdote, I also had a container app fail, from a trusted provider that I assumed weighsy have plenty of testing in place, but maybe it was too synthetic. Sonarr from LinuxServer.io briefly changed to a dev build and boned the DB before switching back to a stable build.
- KronisLV 4y ago
  You know, I thought that I'd actually provide a bit more information about my experiences as well, because while these are anecdotes, they are still something that happened and shouldn't be entirely ignored. These things do happen, sometimes because of the architecture and what's inside of the container in question. For example, that was largely the case with the GitLab Omnibus containers, because of them containing lots of dependencies, or the same with OpenProject. GitLab: https://blog.kronis.dev/everything%20is%20broken/gitlab-updates-are-broken https://blog.kronis.dev/everything%20is%20broken/gitlab-upda... OpenProject: https://blog.kronis.dev/everything%20is%20broken/openproject-updates-are-broken https://blog.kronis.dev/everything%20is%20broken/openproject... (I still think that both are good software, but not without their pain points, like updates in this case) Then again, even something like Nextcloud or comparatively simpler systems like Grav still run into issues with upgrades, sometimes because going across major versions is too much, other times because the containers have vendor dependencies in volumes/bind mounts, which cause a plethora of issues. Nextcloud: https://blog.kronis.dev/everything%20is%20broken/nextcloud-is-broken https://blog.kronis.dev/everything%20is%20broken/nextcloud-i... Grav: https://blog.kronis.dev/everything%20is%20broken/grav-is-broken https://blog.kronis.dev/everything%20is%20broken/grav-is-bro... But it's a little bit odd when ever the operating systems we use have similar issues. Like that mention of a Debian update breaking GRUB prevented the server from booting altogether and that other time the exim4 package getting launched prevented my mail server from starting because of the port being bound. Debian breaking GRUB: https://blog.kronis.dev/everything%20is%20broken/debian-and-grub-are-broken https://blog.kronis.dev/everything%20is%20broken/debian-and-... Debian launching exim4: https://blog.kronis.dev/everything%20is%20broken/debian-updates-are-broken https://blog.kronis.dev/everything%20is%20broken/debian-upda... Other times you get problems with upgrading across libraries, or language runtime versions, which I didn't mention in the original post, but which is no less of an issue. It's even harder than just regular package or software updates, because sometimes it might necessitate a major rewrite. Example of Java 8 to Java 11: https://blog.kronis.dev/everything%20is%20broken/upgrading-java-8-to-11-is-broken https://blog.kronis.dev/everything%20is%20broken/upgrading-j... That said, I'm not sure how good the idea of documenting every software breakage out there is, because the folder with screenshots and information about things that went wrong just seems to be getting longer and longer for me. But hey, maybe it will serve as an insight into what things were around 2020s some day, when hopefully we've figured out how to upgrade software more safely.
linsomniac 4y ago
Agreed, updates are probably at some point going to break things, and you're going to have to spend time fixing them, maybe even recovering from backups... As I said, an operations problem. Leaving things without upgrades is also a problem as well though, due to security problems.
- KronisLV 4y ago
  That is very true, that's also why having (working) backups is so important, or even knowing what to back up and restore, so you don't end up overwriting old vendored packages/dependencies but can restore the configuration/data instead. I'd go as far as to suggest that updating is something that you must do while you have any sort of a stake in the overall outcome of the thing you're working on, given how destructive breaches and getting hacked could be. That also does mean that you will absolutely need to take the operations overhead into account and plan for it.

XCSme 4y ago

I agree, I ran many WordPress sites, UXWizz dashboards or even my own tools written in Node.js, and they never ever broke by themselves in 10+ years. The only time there is an issue if I decide to update/change anything. In general, software that has once worked will always work. If I open my PSP (Playstation Portable) from 2005 it will still work as well as in the first day, all the games work the same and the boot time and interface are faster than most consoles nowadays. Why does it still work? Because it once worked and nothing changed.

dicknuckle 4y ago

Anecdote, I also had a container app fail, from a trusted provider that I assumed weighsy have plenty of testing in place, but maybe it was too synthetic. Sonarr from LinuxServer.io briefly changed to a dev build and boned the DB before switching back to a stable build.

KronisLV 4y ago
You know, I thought that I'd actually provide a bit more information about my experiences as well, because while these are anecdotes, they are still something that happened and shouldn't be entirely ignored. These things do happen, sometimes because of the architecture and what's inside of the container in question. For example, that was largely the case with the GitLab Omnibus containers, because of them containing lots of dependencies, or the same with OpenProject. GitLab: https://blog.kronis.dev/everything%20is%20broken/gitlab-updates-are-broken https://blog.kronis.dev/everything%20is%20broken/gitlab-upda... OpenProject: https://blog.kronis.dev/everything%20is%20broken/openproject-updates-are-broken https://blog.kronis.dev/everything%20is%20broken/openproject... (I still think that both are good software, but not without their pain points, like updates in this case) Then again, even something like Nextcloud or comparatively simpler systems like Grav still run into issues with upgrades, sometimes because going across major versions is too much, other times because the containers have vendor dependencies in volumes/bind mounts, which cause a plethora of issues. Nextcloud: https://blog.kronis.dev/everything%20is%20broken/nextcloud-is-broken https://blog.kronis.dev/everything%20is%20broken/nextcloud-i... Grav: https://blog.kronis.dev/everything%20is%20broken/grav-is-broken https://blog.kronis.dev/everything%20is%20broken/grav-is-bro... But it's a little bit odd when ever the operating systems we use have similar issues. Like that mention of a Debian update breaking GRUB prevented the server from booting altogether and that other time the exim4 package getting launched prevented my mail server from starting because of the port being bound. Debian breaking GRUB: https://blog.kronis.dev/everything%20is%20broken/debian-and-grub-are-broken https://blog.kronis.dev/everything%20is%20broken/debian-and-... Debian launching exim4: https://blog.kronis.dev/everything%20is%20broken/debian-updates-are-broken https://blog.kronis.dev/everything%20is%20broken/debian-upda... Other times you get problems with upgrading across libraries, or language runtime versions, which I didn't mention in the original post, but which is no less of an issue. It's even harder than just regular package or software updates, because sometimes it might necessitate a major rewrite. Example of Java 8 to Java 11: https://blog.kronis.dev/everything%20is%20broken/upgrading-java-8-to-11-is-broken https://blog.kronis.dev/everything%20is%20broken/upgrading-j... That said, I'm not sure how good the idea of documenting every software breakage out there is, because the folder with screenshots and information about things that went wrong just seems to be getting longer and longer for me. But hey, maybe it will serve as an insight into what things were around 2020s some day, when hopefully we've figured out how to upgrade software more safely.

linsomniac 4y ago

Agreed, updates are probably at some point going to break things, and you're going to have to spend time fixing them, maybe even recovering from backups... As I said, an operations problem. Leaving things without upgrades is also a problem as well though, due to security problems.

KronisLV 4y ago
That is very true, that's also why having (working) backups is so important, or even knowing what to back up and restore, so you don't end up overwriting old vendored packages/dependencies but can restore the configuration/data instead. I'd go as far as to suggest that updating is something that you must do while you have any sort of a stake in the overall outcome of the thing you're working on, given how destructive breaches and getting hacked could be. That also does mean that you will absolutely need to take the operations overhead into account and plan for it.