8 ms·

Never Update Anything

185 points by generatorman 2y ago

hitpointdrew 2y ago
"Never Update Anything" Author proceeds to add to two updates to the article, epic troll.
- Joel_Mckay 2y ago
  These days it makes sense to life-cycle entire container images rather than maintain applications with their dependencies. The current BSOD epidemic demonstrated the folly of mass concurrent versioning. *nix admins are used to playing upgrade Chicken with their uptime scores. lol =)
nickthegreek 2y ago
Previously (November 4, 2021 — 319 points, 281 comments): https://news.ycombinator.com/item?id=29106159 https://news.ycombinator.com/item?id=29106159
schiffern 2y ago
Hug of Death right now, mirror here: https://web.archive.org/web/20240509031433/https://blog.kronis.dev/articles/never-update-anything https://web.archive.org/web/20240509031433/https://blog.kron...
- jspash 2y ago
  Maybe they should have updated the server capacity?
  - KronisLV 2y ago
    Currently it's running on a VPS that has 1 CPU core and 4 GB of RAM, resources which are shared with a few other processes. I'm thinking that I might move over from multiple smaller VPSes (better separation of resources, smaller fallout from various issues) to a fewer bigger ones in the future (also cheaper), in which case the containers would struggle less under load spikes.
- cjalmeida 2y ago
  Well, if only he had updated his server stack to something more scalable...
exe34 2y ago
> Not only that, but put anything and everything you will ever need within the standard library or one or two large additional libraries. you can definitely do that with python today: assemble a large group of packages that conver a large fraction of what people need to do, and maintain that as the 1 or 2 big packages. nobody's stopping you.
- ungamedplayer 2y ago
  You would need to maintain python itself too. Imagine if you had done this same plan prior to the python 3 transition.
  - exe34 2y ago
    https://github.com/naftaliharris/tauthon https://github.com/naftaliharris/tauthon it's already being done!
apantel 2y ago
No no no, it’s “never update anything and don’t expose your machine to the internet”. Winning strategy right there.
- 0cf8612b2e1e 2y ago
  If only that were possible with some appliances. I can keep my TV offline, but not the Roku. Internet connected utilities which will continually patch themselves into enshitification.
- KronisLV 2y ago
  I know it's supposed to be a statement to take the absurd title of my article a bit further, but in some cases, I can see that being said unironically. Nothing good would happen if some machine running Windows XP in a hospital that's hooked up to an expensive piece of equipment that doesn't run with anything else suddenly got connected to the Internet. Nor does the idea of any IoT device reaching past the confines of the local network make me feel safe, given how you hear about various exploits that those have. On one hand, you should get security patches whenever possible. On the other hand, it's not realistic to get just security patches with non-breaking changes only. Other times, pieces of hardware and software will just be abandoned (e.g. old Android phones) and then you're on your own, even if you'd want to keep them up to date.
  - vbezhenar 2y ago
    I used to work with state agencies and they run outdated unpatched Windows computers all over the place. Nowadays I work in medical software and hospitals are running outdated unpatched Windows computers everywhere. Nobody cares about updates. Almost nobody. I never saw Windows 11. Windows 10 is popular, but there are plenty of Vistas. I'm outright declining supporting Windows XP and we lost some customers over this issue. My development tools are somewhat outdated, because compilers love to drop old Windows versions and 32-bit architectures, so sometimes I just can't update the compiler. For example I'm stuck with Java 8 for the foreseeable future, because Vista users are too numerous and it's not an option to drop them. Hacker News is like another world. Yes, I update my computer, but everyone else does not. Even my fellow developers often don't care and just use whatever they got.
SoftTalker 2y ago
"In my eyes it could be pretty nice to have a framework version that's supported for 10-20 years and is so stable that it can be used with little to no changes for the entire expected lifetime of a system." This is what applications used to be like, before the web and internet hit and regular or even push updating became easy. It was simply so difficult and expensive to provide updates once the software was in the customer's hands that it was done intentionally and infrequently. For the most part, you bought software, installed it, and used it. That was it. It never changed, unless you bought it again in a newer version
- EvanAnderson 2y ago
  Frequent updates, in the old days, meant that a vendor had poor QA. I think that's probably still the case most of the time today, too.
  - hulitu 2y ago
    See Microsoft and Crowdstrike for details. /s
  - tivert 2y ago
    > Frequent updates, in the old days, meant that a vendor had poor QA. I think that's probably still the case most of the time today, too. The internet has normalized poor QA. The bosses don't give a shit about QA anymore because it's so cheap to just push out a patch. I mean just look at old video game magazines that talked about the software development process: the developers would test the hell out of a game, then test the hell out of it again, because once it was burned onto a $100 cart (in 2024 dollars) it wasn't ever going to change. Now games can remain buggy and unstable for months or even years after "release."
    - dasil003 2y ago
      I never worked on games, but I did do a streaming video app for PS3 in 2010, during the time period when it was arguably the best media center box available. Working with Sony (SCEE) on this was eye opening how their QA process was set up. Essentially you formally submitted the compiled app (C++ and Flash!) to them, and then tickets would come back and you'd have to address them. QA was a separate org so you never really met them in person, all issue discussion would happen in-band to the ticketing system. QA had all the power to decide when and if this thing would ship, much moreso than Product or Engineering. I can't say the process made a ton of sense for a downloadable app powered by a multi-platform online service, but it was illuminating as to how the high quality bar of 90s/00s console games was achieved.
spyspy 2y ago
Kinda weird to see Java over Go, when the former is basically an entirely new language from what it was 10 years ago and the latter has made it an explicit goal to never break older versions and (almost) never change the core language.
- karolist 2y ago
  Writing backends in Go I do get that warm fuzzy feeling knowing that it will compile and work in ten years. The syntax is easy to read, if I'm not lazy to add extensive tests I can simply read these as documentation to re-familiarise myself later. It's now my go to tool for everything server side.
  - ungamedplayer 2y ago
    Now do it for lisp and your libraries alone were last updated 5 years ago.
KronisLV 2y ago
Oh hey, I was wondering why the VPS suddenly had over 100 load average, restarted Docker since the containers were struggling, now I know why (should be back now for a bit). Won't necessarily fix it, might need to migrate over to something else for the blog, with a proper cache, alongside actually writing better articles in the future. I don't think the article itself holds up that well, it's just that updates are often a massive pain, one that you have to deal with somehow regardless. Realistically, LTS versions of OS distros and technologies that don't change often will lessen the pain, but not eliminate it entirely. And even then, you'll still need to deal with breaking changes when you will be forced to upgrade across major releases (e.g. JDK 8 to something newer after EOL) or migrate once a technology dies altogether (e.g. AngularJS). It's not like people will backport fixes for anything indefinitely either.
- cjalmeida 2y ago
  >might need to migrate over to something else for the blog, with a proper cache Never Update _Anything_ :)
  - KronisLV 2y ago
    I am very much tempted not to because it works under lower loads, could just put it on a faster server, but how could I pass up the chance to write my own CMS (well, a better one than the previous ones I've done)? That's like a rite of passage. But yes, the irony isn't lost on me, I just had to go for that title.
    - stavros 2y ago
      If you have to write your own CMS, make it compile to static files. I did that with Django, used Django-distill, and it's hands down the best static site generator I've ever used. My site never needs updates and never goes down under any amount of load.
      - hipadev23 2y ago
        “static files” are nothing more than no-TTL caching strategy with manual eviction.
cogman10 2y ago
I find it pretty funny that immediately on the first click of this article I was greeted with an internal server error.
- KronisLV 2y ago
  That was me scrambling to allocate more resources to the container and redeploy it, after my alerting ticked me off about issues and I figured out what's going on. While the container itself was down, the reverse proxy returned an error.
neontomo 2y ago
the react module bloat example is not a fair one, the recommended way to start a react project isn't to use create-react-app. other methods are more streamlined. but then again, the deprecation of create-react-app perhaps proves the point that updates create problems.
- eXpl0it3r 2y ago
  It's not anymore the recommended way and last I checked it's not really being maintained as much as other ways, but for quite a while, it was the recommended way.
  - neontomo 2y ago
    that is what i'm saying ;-)
AaronFriel 2y ago
A feature I've wanted for ages, for every OS package manager (Windows, apt, yum, apk, etc.), every language's package manager (npm, pypi, etc.), and so on is to update but filter out anything less than one day, one week, or one month old. And it applies here, too. Now, some software, they effectively do this risk mitigation for you. Windows, macOS, browsers all do this very effectively. Maybe only the most cautious enterprises delay these updates by a day. But even billion dollar corporations don't do a great job of rolling out updates incrementally. This especially applies as tools exist to automatically scan for dependency updates, the list of these is too long to name - don't tell me about an update only a day old, that's too risky for my taste. So for OS and libraries for my production software? I'm OK sitting a week or a month behind, let the hobbyists and the rest of the world test that for me. Just give me that option, please.
- n_plus_1_acc 2y ago
  Isn't that basically non rolling-release distros?
  - c0balt 2y ago
    Yesn't, many still release updates frequently as long (usually, if server etc.) as they are compatible. Mostly though only minor updates for features. This is required for some components, like, e.g., glibc or openssh, to stay secure-ish.
- gnramires 2y ago
  Debian has 2/3 stages of software deployment that I know of: Unstable, Testing and Stable. By the time it comes to stable it has been quite extensively tested. The exceptions are only security updates which you may want to get very quickly anyway. I really recommend Debian (in particular with unattended security upgrades) for severs. Other distros have this as well (Thumbleweed, Void, etc.), and I really think most people should not be using recently-deployed software. A small community using them however helps testing so the rest of us can have more stability. Which is why I don't recommend using Arch (or Debian unstable) for general users, unless you specifically want to help testing and accept the risk. Also randomizing update schedules by at least a few hours does seem very wise (I don't think even the most urgent updates would make or break in say 6 hours of randomization?)
knallfrosch 2y ago
We're being paid to migrate our hardware boxes programmatically to Windows 10 IoT LTSC so that new boxes ship with 10+ years of security. We're still supporting some XP devices (not connected to the internet.) So to anyone depending on us: You're welcome. But let me tell you something: Long-Term Support software mostly doesn't pay well, and it's not fun either. Meanwhile some Google clown is being paid 200k to fuck up Fitbit or rewrite Wallet for the 5th time in the newest language. So yeah. I'd love to have stable, reliable dependencies while I'm mucking around with the newest language de jour. But you see how that doesn't work, right?
- kccqzy 2y ago
  The fucking up of Fitbit and the rewriting of Wallet are not the engineers' fault. These kind of projects are mostly decided and planned by PMs: clueless and incompetent PMs. For payments in particular it was not even just an incompetent PM, but an incompetent director that saw the success of the NBU Paisa payment in India and thought the U.S. would be the same. The engineers are at most just complicit. Those who aren't are laid off or they quit on their own accord.
  - ozim 2y ago
    Engineers are mostly complicit because they get that 200k$ salary when they chase next shiny thing. No one is paying such salaries for mundane clerical job.
  - chrisjj 2y ago
    You're saying complicity is not a fault?? :)
  - binary132 2y ago
    Lead engineers and architects chasing resume-filler projects are absolutely part of the problem
msoad 2y ago
Kinda ironic that the article itself was updated
CooCooCaCha 2y ago
I disagree, keep things constantly updated (within reason). Most companies I've worked for have the attitude of the author, they treat updates as an evil that they're forced to do occasionally (for whatever reason) and, as a result, their updates are more painful than they need to be. It's a self-fulfilling prophecy.
eXpl0it3r 2y ago
Even as a developer not focused on web dev this sounds pretty bad, unless everyone in your dependency tree (from OS to language to libraries) decides to make a switch and even then, you'll be stuck with outdated ways to do things. Who wants to continue maintaining C++03 code bases without all the C++11/14/17/20 features? Who wants to continue using .NET Framework, when all the advances are made in .NET? Who wants to be stuck with libraries full of vulnerabilities and who accepts the risk? Not really addressed is the issue of developers switching jobs/projects every few years. Nobody is sticking around long enough to amass the knowledge needed to ensure maintenance of any larger code base. Which is caused by or caused the companies to also not commit themselves for any longer period of times. If the company expects people to leave within two years and doesn't put in the monetary and non-monetary effort to retain people, why should devs consider anything longer than the current sprint?
- KronisLV 2y ago
  > Who wants to continue maintaining C++03 code bases without all the C++11/14/17/20 features? Who wants to continue using .NET Framework, when all the advances are made in .NET? Who wants to be stuck with libraries full of vulnerabilities and who accepts the risk? With the exception that in this hypothetical world we'd get backported security updates (addressing that particular point), who'd want something like this would be the teams working on large codebases that: - need to keep working in the future and still need to be maintained - are too big or too time consuming to migrate to a newer tech stack (with breaking changes in the middle) with the available resources - are complex in of themselves, where adding new features could be a detriment (e.g. different code styles, more things to think about etc.) Realistically, that world probably doesn't exist and you'll be dragged kicking and screaming into the future, once your Spring version hits EOL (or worse yet, will work with unsupported old versions and watch the count of CVEs increase, hopefully very few will find themselves in this set of circumstances). Alternatively, you'll just go work somewhere else and it'll be someone else's problem, since there are plenty of places where you'll always try to keep things up to date as much as possible, so that the delta between any two versions of your dependencies will be manageable, as opposed to needing to do "the big rewrite" at some point. That said, enterprises already often opt for long EOL Linux distros like RHEL and there is a lot of software out there that is stuck on JDK 8 (just a very visible example) with no clear path of what to do once it reaches EOL, so it's not like issues around updates don't exist. Then again, not a lot of people out there need to think about these things, because the total lifetime of any given product, project, their tenure in the org or even the company itself might not be long enough for those issues to become that apparent.
pron 2y ago
Because of this, in the JDK we've adopted a model we call "tip & tail". The idea is that there are multiple release trains, but they're done in such a way that 1/ different release trains target different audiences and 2/ the model is cheap to maintain -- cheaper, in fact, than many others, especially that of a single release train. The idea is to realise that there are two different classes of consumers who want different things, and rather than try to find a compromise that would not fully satisfy either group (and turns out to be more expensive to boot), we offer multiple release trains for different people. One release train, called the tip, contains new features and performance enhancements in addition to bug fixes and security patches. Applications that are still evolving can benefit from new features and enhancements and have the resources to adopt them (by definition, or else they wouldn't be able to use the new features). Then there are multiple "tail" release trains aimed at applications that are not interested in new features because they don't evolve much anymore (they're "legacy"). These applications value stability over everything else, which is why only security patches and fixes to the most severe bugs are backported to them. This also makes maintaining them cheap, because security patches and major bugs are not common. We fork off a new tail release train from the tip every once in a while (currently, every 2 years). Some tail users may want to benefit from performance improvements and are willing to take the stability risk involved in having them backported, but they can obviously live without them because they have so far. If their absence were painful enough to justify increasing their resources, they could invest in migrating to a newer tail once. Nevertheless, we do offer a "tail with performance enhancements" release train in special circumstances (if there's sufficient demand) -- for pay. The challenge is getting people to understand this. Many want a particular enhancement they personally need backported, because they think that a "patch" with a significant enhancement is safer than a new feature release. They've yet to internalise that what matters isn't how a version is called (we don't use semantic versioning because we think it is unhelpful and necessarily misleading), but that there's an inherent tension between enhancements and stability. You can get more of one or the other, but not both.
- deleted 2y ago
  [deleted]
- jldugger 2y ago
  Is this any different than the LTS approach Canonical and others take?
ricksunny 2y ago
The 'Skip this Update [pro]' button example (Docker Desktop) just made me facepalm and helped me internalize that I'm not a luddite from technology, I'm a luddite from the collectives of people (not the individual people...(!) ) feeling compelled to craft these dark business patterns.
dzonga 2y ago
java over golang lol. when golang has literally been version stable for over a decade now
- canpan 2y ago
  I think the article is interesting. But you are right, golang is the one language I would recommend to someone who does not like updates.
cupantae 2y ago
I’ve supported enterprise software for various big companies and I can tell you that most decision makers for DCs agree with this sentiment. EMC had a system called Target Code which was typically the last patch in the second-last family. But only after it had been in use for some months and/or percentage of customer install base. It was common sense and customers loved it. You don’t want your storage to go down for unexpected changes. Dell tried to change that to “latest is target” and customers weren’t convinced. Account managers sheepishly carried on an imitation of the old better system. Somehow from a PR point of view, it’s easier to cause new problems than let the known ones occur.
_yb2s 2y ago
I pretty much agree- most systems don't need updating. I've seen and setup OpenBSD servers that ran for a decade without issues never getting updates. I currently run some production web services on Debian where I do updates every 3 years or so, and no issues. Leaving something alone that works good is a good strategy. Most of the cars on the road are controlled by ECUs that have never had, and never will have any type of updates, and that is a good thing. Vehicles that can get remote updates like Teslas are going to be much less reliable than one not connected to anything that has a single extensively tested final version. An OS that is fundamentally secure by design, and then locked down to not do anything non-essential, doesn't really need updates unless, e.g. it is a public facing web server, and the open public facing service/port has a known remote vulnerability, which is pretty rare.
- hcarvalhoalves 2y ago
  > Vehicles that can get remote updates like Teslas are going to be much less reliable than one not connected to anything that has a single extensively tested final version. I don’t think it’s necessarily this, but the fact that being able to update anytime is a great source of pressure to release untested software at any cost.
albertP 2y ago
He. I've been running always everything one or two versions behind latest (for my personal laptop, not servers). That means mainly OS (e.g., macOS), but as long as I can avoid automatic updates, I do so. I believe the chances of having a bricked laptop because of a bad update are higher than the chances of getting malware because running one or two versions behind the latest one.
Plasmoid 2y ago
I was working at a place that delivered onprem software. One customer asked us "We like features of version N but we're running N-1. Can you backport them so you don't have to upgrade?". I replied we'd already done that, it was called version N.
- yjftsjthsd-h 2y ago
  I mean... there could be a case for "does this feature really require breaking changes?" that would distinguish those.
mike741 2y ago
Urgent updates can be necessary every once in a while but should be recognized as technical failures on the part of the developers. Failure can be forgiven, but only so many times. The comments saying "what about X update that had this feature I need?" are missing the point entirely. Instead ask yourself about all of the updates you've made without even looking at the patch notes, because there are just too many updates and not enough time. Instead of blaming the producers for creating a blackbox relationship with the consumers, we blame the consumer and blindly tell them to "just update." That's what needs to change. It's a bit similar to opaque ToS issues.
ghawk1ns 2y ago
There is humor in this blog itself has 2 updates
userbinator 2y ago
There is also another type of update: security updates that don't actually matter in the environment that the software is used in. The question of whether the "new features" are for or against the user is another point to ponder.
cpncrunch 2y ago
A reasonable strategy is to wait a week after release before applying an update, unless it's a zero day fix.
palata 2y ago
> My experience shows that oftentimes these new and supposedly backwards compatible features still break old functionality. Well that's the first issue: downright malpractice. Developers should learn how to know (and test) whether it is a major change or not. The current situation is that developers mostly go "YOLO" with semantic versioning and then complain that it doesn't work. Of course it doesn't work if we do it wrong.
palata 2y ago
What the article points to is that most updates are bad updates. We teach people that they should accept all updates for security reasons, but really they should only accept security updates. But they can't, because this is not a possibility that is given to them. All updates are put together, and we as an industry suck at even knowing if our change is backward compatible or not (which is actually some kind of incompetence). And of course it's hard, because users are not competent enough to distinguish good software from bad software, so they follow what the marketing tells them. Meaning that even if you made good software with fewer shiny features but actual stability, users would go for the worse software of the competitor, because it has the latest damn AI buzzword. Sometimes I feel like software is pretty much doomed: it won't get any better. But one thing I try to teach people is this: do not add software to things that work, EVER. You don't want a connected fridge, a connected light bulb or a connected vacuum-cleaner-camera-robot. You don't need it; it's completely superfluous. Also for things that actually matter, many times you don't want them either. Electronic voting is an example I have in my mind: it's much easier to hack a computer from anywhere in the world than to hack millions of pieces of paper.
- hulitu 2y ago
  > And of course it's hard, because users are not competent enough to distinguish good software from bad software, There was a time when Windows had a description for updates. Now the only distinction is between KB3587690 and KB67457770.
- o11c 2y ago
  I also noticed that there's another kind of update not mentioned: * Updates that add a theoretically independent feature, but which other software will dynamically detect and change their behavior for, so that it's not actually independent.
- ranger207 2y ago
  And you can't have _just_ security updates, because the combinational complexity of security fixes across feature versions is insane, not to mention the fact that the interaction of the security and features changes can themselves introduce bugs. There's groups that try (eg distro maintainers) but it's ultimately a losing battle. I'm convinced that patching is only a bandaid, but it's also impossible to have 100% bug-free code, so there needs to be some sort of systematic solution on top of whatever particular code is running. Behavior analysis, egress network analysis, immutable by default images with strictly defined writeable volumes and strictly defined data that's going to be written there, etc. There's not a silver bullet, but I think patching and trying to keep up with updates is, like, a gallium bullet at best
tonymet 2y ago
our industry could use a risk assessment index scanner on updates, similar to "npm audit" , that measures the delta between versions and gives a risk indicator based on a number of parameters. The issue with changelogs is that they are an honor system, and they don't objectively assess the risk of the update. Comparing changes in the symbol table and binary size could give a reasonable red/yellow/green indicator of the risk of an update. Over time, you could train a classifier to give even more granularity and confidence.
k_roy 2y ago
> In my eyes it could be pretty nice to have a framework version that's supported for 10-20 years and is so stable that it can be used with little to no changes for the entire expected lifetime of a system. Yeah, me too. I also would like a few million bucks in the bank. It's naive to think that every project wouldn't want to set this goal, simply because it's so unrealistic.
walterbell 2y ago
> And lastly, choose boring technology. Only use software that you're sure will be supported in 10 years. Only use software, which has very few "unknown unknowns". Only use software, where the development pace has slowed down to a reasonable degree. Perl has been stable for a couple of decades.
- iso8859-1 2y ago
  Perl 6 is incompatible in all sorts of ways. It's so incompatible that people didn't even upgrade! So incompatible that they renamed the language! Even more incompatible than Python 3! Even Perl 5 is rapidly evolving. They just added try/catch. Added a new isa operator. Added a new __CLASS__ keyword. Added defer blocks.
binary132 2y ago
I had started to think I was the only one saying this.
gtirloni 2y ago
The world is not static and software these days is very interconnected. Dreams of not updating only work in a unchanging world. Sadly, this world is still to be found.
- LargoLasskhyfv 2y ago
  Funny you're saying that. In context of the stricken crowds I've read that FedEx, UPS and SouthWest had no problems at all, because many systems run on Windows95, or even something based on Win3.1x :-) Seems they are well isolated, but networked nonetheless. Or do you think they use bush-drums and smoke signs to conduct their businesses?
  - Groxx 2y ago
    That probably has more to do with not jumping on the latest fad* and installing an unattended unrestricted automatically-globally-updating rootkit on all of your critical machines. That is generally recognizable as stupidity. And the ones that did so are now paying the price. * compliance tactics are very prone to fads. just look at cookie banners.
hcarvalhoalves 2y ago
Extreme viewpoint, but agree strongly. Big reason why working in Common Lisp brings a smile to my face - it’s a standard, quicklisp works, ffi works, etc. I can run code and follow instructions written DECADES ago, it just damn works.
1vuio0pswjnm7 2y ago
Avoid software that will need constant updates. Because that is a signal it is defective to begin with, or expected to be broken soon. For example, I avoid graphical commercial OS, large, graphical web browsers. Especially mobile OS and "apps". Avoidance does not have to be 100% to be useful. If it defeats reliance on such software then it pays for itself, so to speak. IMHO. The notion of allowing RCE/OTA for "updates" might allegedly be motivated by the best of intentions. But these companies are not known for their honesty. Nor for benevolence. And let's be honest, allowing remote access to some company will not be utilised 100% for the computer owner's benefit. For the companies remotely installing and automatically running code on other peoples' computer, surveillance has commercial value. Allowing remote access makes surveillance easier. A cake walk.
- m3kw9 2y ago
  Exception is mobile apps
Groxx 2y ago
To jump on a related article since it's linked and comments are now closed: https://blog.kronis.dev/articles/stable-software-release-system https://blog.kronis.dev/articles/stable-software-release-sys... The 2021/2022/2023/2024 version-numbering schemes are for applications, not libraries, because applications are essentially not ever semver-stable. That's perfectly reasonable for them. They don't need semver. People don't build against jetbrains-2024.1, they just update their stuff when JetBrains breaks something they use (which can happen at literally any time, just ask plugin devs)... because they're fundamentally unstable products and they don't care about actual stability, they just do an okay job and call it Done™ and developers on their APIs are forced to deal with it. Users don't care 99%+ of the time because the UI doesn't change and that is honestly good enough in nearly all cases. That isn't following semver, which is why they don't follow semver. Which is fine (because they control their ecosystem with an iron fist). It's a completely different relationship with people looking at that number. For applications, I totally agree. Year-number your releases, it's much more useful for your customers (end-users) who care about if their habits are going to be interrupted and possibly how old it is. But don't do it with libraries, it has next to nothing to do with library customers (developers) who are looking for mechanical stability.
kkfx 2y ago
Or use NixOS/Guix Systems instead of living in the stone age of containers...
- k_roy 2y ago
  Or, alternatively, realize both are valid options and have different use-cases. Or big-brain it and use Nix to build your containers and get the best of both worlds.
- notfed 2y ago
  I ran into critical issues with some NixOS packages I needed, the maintainer doesn't seem to care about my use case, and I can't for the life of me understand the Nix language enough to fix it myself. Have stopped using NixOS because of that. If Nix language was replaced with something sensible I'd jump back in excitedly.
  - kkfx 2y ago
    The point is not much the project per se but the concept, Infrastructure as Code, OS as code. So far NixOS is a relatively small ecosystem, in a complex community situation with the main sponsors, a mil-tech company, who have got a wave of backslash for debatable reasons, Guix System on other side it's mostly a French INRIA project focused more on HPC than anything else, so both have their corners, but they show a modern way to develop and deploy OSes as OpenSolaris (since it was before IllumOS fork) show the first integration of storage (zfs) and package system (IPS) and bootloader (with the boot environment concept), somewhat badly copied later by FreeBSD and partially on GNU/Linux with a set of scripts. ALL OTHER OSes/distro are still stuck in the '80s in that sense and most people seems even unable to understand. On storage alone the famous "rampant layer violation" and the absurdity of btrfs and stratis "against" zfs are really good examples of blind tech reactionary behaviors by high skilled people and their outcomes a showcase of why we damn need to innovate instead of shooting ourselves in the feet switching from something obsolete to something even worse (like the now-almost-finished full stack virtualization on x86 and thereafter the paravirtualization/container mania still current) layering crap on crap with more and more unmanageable infra with enormous attack surfaces. Another small and relatively known example: Home Assistant project: apart of their design, they choose to distribute a python application as a GNU/Linux entire distro because to the such move seems to be commercially sound and many others choose to follow them instead of simply pip-install HA in a local venv, wasting an immense amount of resources on their system for what? Such kind of tech evolution must end or we will collapse soon digitally speaking.
aspyct 2y ago
There's an alternative solution: update everything, but limit your dependencies. Example: for my (personal) projects, I only use whatever is available in the debian repositories. If it's not in there, it's not on my dependency list. Then enable unattended upgrades, and forget about all that mess.
mikewarot 2y ago
When I was the sole IT guy for a small consulting company, once I got everything working, I never updated it. We used Microsoft office 2000 for 12 years. Never had to retrain people, deal with the weird ribbon toolbar, etc. It's only the deranged use of OSs with ambient authority that gums up what would otherwise be stable systems.
msiemens 2y ago
I know it's this is a rather long tangent and not the main point of the article, but regarding "Docker Swarm over Kubernetes", I've had a ton of bad experiences at my employer running a production Swarm cluster. Among them: - Docker Swarm and Docker Compose use different parsers for `docker-compose.yaml` files, which may lead to the same file working with Compose but not with Swarm ([1]). - A Docker network only supports up to 128 joined containers (at least when using Swarm). This is due to the default address space for a Docker network using a /24 network (which the documentation only mentions in passing). But, Docker Swarm may not always show error message indicating that it's a network problem. Sometimes services would just stay in "New" state forever without any indictation what's wrong (see e.g. [2]). - When looking a up a service name, Docker Swarm will use the IP from the first network (sorted lexically) where the service name exists. In a multi-tenant setup, where a lot of services are connected to an ingress network (i.e. Taefik), this may lead to a service connecting to a container from a different network than expected. The only solution is to always append the network name to the service name (e.g. service.customer-network; see [3]). - Due to some reason I still wasn't able to figure out, the cluster will sometimes just break. The leader loses its connection to the other manager nodes, which in turn do NOT elect a new leader. The only solution is to force-recreate the whole cluster and then redeploy all workloads (see [4]). Sure, our use case is somewhat special (running a cluster used by a lot of tenants), and we were able to find workarounds (some more dirty than others) to most of our issues with Docker Swarm. But what annoys me is that for almost all of the issues we had, there was a GitHub ticket that didn't get any official response for years. And in many cases, the reporters just give up waiting and migrate to K8s out of despair or frustration. Just a few quotes from the linked issues: > We, too, started out with Docker Swarm and quickly saw all our production clusters crashing every few days because of this bug. […] This was well over two years (!) ago. This was when I made the hard decision to migrate to K3s. We never looked back. > We recently entirely gave up on Docker Swarm. Our new cluster runs on Kubernetes, and we've written scripts and templates for ourselves to reduce the network-stack management complexities to a manageable level for us. […] In our opinion, Docker Swarm is not a production-ready containerization environment and never will be. […] Years of waiting and hoping have proved fruitless, and we finally had to go to something reliable (albeit harder to deal with). > IMO, Docker Swarm is just not ready for prime-time as an enterprise-grade cluster/container approach. The fact that it is possible to trivially (through no apparent fault of your own) have your management cluster suddenly go brainless is an outrage. And "fixing" the problem by recreating your management cluster is NOT a FIX! It's a forced recreation of your entire enterprise almost from scratch. This should never need to happen. But if you run Docker Swarm long enough, it WILL happen to you. And you WILL plunge into a Hell the scope of which is precisely defined by the size and scope of your containerization empire. In our case, this was half a night in Hell. […] This event was the last straw for us. Moving to Kubernetes. Good luck to you hardy souls staying on Docker Swarm! Sorry, if this seems like like Docker Swarm bashing. K8s has it's own issues, for sure! But at least there is a big community to turn to for help, if things to sideways. [1]: https://github.com/docker/cli/issues/2527 https://github.com/docker/cli/issues/2527 [2]: https://github.com/moby/moby/issues/37338 https://github.com/moby/moby/issues/37338 [3]: https://github.com/docker/compose/issues/8561#issuecomment-1873501393 https://github.com/docker/compose/issues/8561#issuecomment-1... [4]: https://github.com/moby/moby/issues/34384 https://github.com/moby/moby/issues/34384

185 points by generatorman 2y ago

hitpointdrew 2y ago

"Never Update Anything" Author proceeds to add to two updates to the article, epic troll.

Joel_Mckay 2y ago
These days it makes sense to life-cycle entire container images rather than maintain applications with their dependencies. The current BSOD epidemic demonstrated the folly of mass concurrent versioning. *nix admins are used to playing upgrade Chicken with their uptime scores. lol =)

nickthegreek 2y ago

Previously (November 4, 2021 — 319 points, 281 comments): https://news.ycombinator.com/item?id=29106159 https://news.ycombinator.com/item?id=29106159

schiffern 2y ago

Hug of Death right now, mirror here: https://web.archive.org/web/20240509031433/https://blog.kronis.dev/articles/never-update-anything https://web.archive.org/web/20240509031433/https://blog.kron...

jspash 2y ago
Maybe they should have updated the server capacity?
- KronisLV 2y ago
  Currently it's running on a VPS that has 1 CPU core and 4 GB of RAM, resources which are shared with a few other processes. I'm thinking that I might move over from multiple smaller VPSes (better separation of resources, smaller fallout from various issues) to a fewer bigger ones in the future (also cheaper), in which case the containers would struggle less under load spikes.
cjalmeida 2y ago
Well, if only he had updated his server stack to something more scalable...

exe34 2y ago

> Not only that, but put anything and everything you will ever need within the standard library or one or two large additional libraries. you can definitely do that with python today: assemble a large group of packages that conver a large fraction of what people need to do, and maintain that as the 1 or 2 big packages. nobody's stopping you.

ungamedplayer 2y ago
You would need to maintain python itself too. Imagine if you had done this same plan prior to the python 3 transition.
- exe34 2y ago
  https://github.com/naftaliharris/tauthon https://github.com/naftaliharris/tauthon it's already being done!

apantel 2y ago

No no no, it’s “never update anything and don’t expose your machine to the internet”. Winning strategy right there.

0cf8612b2e1e 2y ago
If only that were possible with some appliances. I can keep my TV offline, but not the Roku. Internet connected utilities which will continually patch themselves into enshitification.
KronisLV 2y ago
I know it's supposed to be a statement to take the absurd title of my article a bit further, but in some cases, I can see that being said unironically. Nothing good would happen if some machine running Windows XP in a hospital that's hooked up to an expensive piece of equipment that doesn't run with anything else suddenly got connected to the Internet. Nor does the idea of any IoT device reaching past the confines of the local network make me feel safe, given how you hear about various exploits that those have. On one hand, you should get security patches whenever possible. On the other hand, it's not realistic to get just security patches with non-breaking changes only. Other times, pieces of hardware and software will just be abandoned (e.g. old Android phones) and then you're on your own, even if you'd want to keep them up to date.
- vbezhenar 2y ago
  I used to work with state agencies and they run outdated unpatched Windows computers all over the place. Nowadays I work in medical software and hospitals are running outdated unpatched Windows computers everywhere. Nobody cares about updates. Almost nobody. I never saw Windows 11. Windows 10 is popular, but there are plenty of Vistas. I'm outright declining supporting Windows XP and we lost some customers over this issue. My development tools are somewhat outdated, because compilers love to drop old Windows versions and 32-bit architectures, so sometimes I just can't update the compiler. For example I'm stuck with Java 8 for the foreseeable future, because Vista users are too numerous and it's not an option to drop them. Hacker News is like another world. Yes, I update my computer, but everyone else does not. Even my fellow developers often don't care and just use whatever they got.

SoftTalker 2y ago

"In my eyes it could be pretty nice to have a framework version that's supported for 10-20 years and is so stable that it can be used with little to no changes for the entire expected lifetime of a system." This is what applications used to be like, before the web and internet hit and regular or even push updating became easy. It was simply so difficult and expensive to provide updates once the software was in the customer's hands that it was done intentionally and infrequently. For the most part, you bought software, installed it, and used it. That was it. It never changed, unless you bought it again in a newer version

EvanAnderson 2y ago
Frequent updates, in the old days, meant that a vendor had poor QA. I think that's probably still the case most of the time today, too.
- hulitu 2y ago
  See Microsoft and Crowdstrike for details. /s
- tivert 2y ago
  > Frequent updates, in the old days, meant that a vendor had poor QA. I think that's probably still the case most of the time today, too. The internet has normalized poor QA. The bosses don't give a shit about QA anymore because it's so cheap to just push out a patch. I mean just look at old video game magazines that talked about the software development process: the developers would test the hell out of a game, then test the hell out of it again, because once it was burned onto a $100 cart (in 2024 dollars) it wasn't ever going to change. Now games can remain buggy and unstable for months or even years after "release."
  - dasil003 2y ago
    I never worked on games, but I did do a streaming video app for PS3 in 2010, during the time period when it was arguably the best media center box available. Working with Sony (SCEE) on this was eye opening how their QA process was set up. Essentially you formally submitted the compiled app (C++ and Flash!) to them, and then tickets would come back and you'd have to address them. QA was a separate org so you never really met them in person, all issue discussion would happen in-band to the ticketing system. QA had all the power to decide when and if this thing would ship, much moreso than Product or Engineering. I can't say the process made a ton of sense for a downloadable app powered by a multi-platform online service, but it was illuminating as to how the high quality bar of 90s/00s console games was achieved.

spyspy 2y ago

Kinda weird to see Java over Go, when the former is basically an entirely new language from what it was 10 years ago and the latter has made it an explicit goal to never break older versions and (almost) never change the core language.

karolist 2y ago
Writing backends in Go I do get that warm fuzzy feeling knowing that it will compile and work in ten years. The syntax is easy to read, if I'm not lazy to add extensive tests I can simply read these as documentation to re-familiarise myself later. It's now my go to tool for everything server side.
- ungamedplayer 2y ago
  Now do it for lisp and your libraries alone were last updated 5 years ago.

KronisLV 2y ago

Oh hey, I was wondering why the VPS suddenly had over 100 load average, restarted Docker since the containers were struggling, now I know why (should be back now for a bit). Won't necessarily fix it, might need to migrate over to something else for the blog, with a proper cache, alongside actually writing better articles in the future. I don't think the article itself holds up that well, it's just that updates are often a massive pain, one that you have to deal with somehow regardless. Realistically, LTS versions of OS distros and technologies that don't change often will lessen the pain, but not eliminate it entirely. And even then, you'll still need to deal with breaking changes when you will be forced to upgrade across major releases (e.g. JDK 8 to something newer after EOL) or migrate once a technology dies altogether (e.g. AngularJS). It's not like people will backport fixes for anything indefinitely either.

cjalmeida 2y ago
>might need to migrate over to something else for the blog, with a proper cache Never Update _Anything_ :)
- KronisLV 2y ago
  I am very much tempted not to because it works under lower loads, could just put it on a faster server, but how could I pass up the chance to write my own CMS (well, a better one than the previous ones I've done)? That's like a rite of passage. But yes, the irony isn't lost on me, I just had to go for that title.
  - stavros 2y ago
    If you have to write your own CMS, make it compile to static files. I did that with Django, used Django-distill, and it's hands down the best static site generator I've ever used. My site never needs updates and never goes down under any amount of load.
    - hipadev23 2y ago
      “static files” are nothing more than no-TTL caching strategy with manual eviction.

cogman10 2y ago

I find it pretty funny that immediately on the first click of this article I was greeted with an internal server error.

KronisLV 2y ago
That was me scrambling to allocate more resources to the container and redeploy it, after my alerting ticked me off about issues and I figured out what's going on. While the container itself was down, the reverse proxy returned an error.

neontomo 2y ago

the react module bloat example is not a fair one, the recommended way to start a react project isn't to use create-react-app. other methods are more streamlined. but then again, the deprecation of create-react-app perhaps proves the point that updates create problems.

eXpl0it3r 2y ago
It's not anymore the recommended way and last I checked it's not really being maintained as much as other ways, but for quite a while, it was the recommended way.
- neontomo 2y ago
  that is what i'm saying ;-)

AaronFriel 2y ago

A feature I've wanted for ages, for every OS package manager (Windows, apt, yum, apk, etc.), every language's package manager (npm, pypi, etc.), and so on is to update but filter out anything less than one day, one week, or one month old. And it applies here, too. Now, some software, they effectively do this risk mitigation for you. Windows, macOS, browsers all do this very effectively. Maybe only the most cautious enterprises delay these updates by a day. But even billion dollar corporations don't do a great job of rolling out updates incrementally. This especially applies as tools exist to automatically scan for dependency updates, the list of these is too long to name - don't tell me about an update only a day old, that's too risky for my taste. So for OS and libraries for my production software? I'm OK sitting a week or a month behind, let the hobbyists and the rest of the world test that for me. Just give me that option, please.

n_plus_1_acc 2y ago
Isn't that basically non rolling-release distros?
- c0balt 2y ago
  Yesn't, many still release updates frequently as long (usually, if server etc.) as they are compatible. Mostly though only minor updates for features. This is required for some components, like, e.g., glibc or openssh, to stay secure-ish.
gnramires 2y ago
Debian has 2/3 stages of software deployment that I know of: Unstable, Testing and Stable. By the time it comes to stable it has been quite extensively tested. The exceptions are only security updates which you may want to get very quickly anyway. I really recommend Debian (in particular with unattended security upgrades) for severs. Other distros have this as well (Thumbleweed, Void, etc.), and I really think most people should not be using recently-deployed software. A small community using them however helps testing so the rest of us can have more stability. Which is why I don't recommend using Arch (or Debian unstable) for general users, unless you specifically want to help testing and accept the risk. Also randomizing update schedules by at least a few hours does seem very wise (I don't think even the most urgent updates would make or break in say 6 hours of randomization?)

knallfrosch 2y ago

We're being paid to migrate our hardware boxes programmatically to Windows 10 IoT LTSC so that new boxes ship with 10+ years of security. We're still supporting some XP devices (not connected to the internet.) So to anyone depending on us: You're welcome. But let me tell you something: Long-Term Support software mostly doesn't pay well, and it's not fun either. Meanwhile some Google clown is being paid 200k to fuck up Fitbit or rewrite Wallet for the 5th time in the newest language. So yeah. I'd love to have stable, reliable dependencies while I'm mucking around with the newest language de jour. But you see how that doesn't work, right?

kccqzy 2y ago
The fucking up of Fitbit and the rewriting of Wallet are not the engineers' fault. These kind of projects are mostly decided and planned by PMs: clueless and incompetent PMs. For payments in particular it was not even just an incompetent PM, but an incompetent director that saw the success of the NBU Paisa payment in India and thought the U.S. would be the same. The engineers are at most just complicit. Those who aren't are laid off or they quit on their own accord.
- ozim 2y ago
  Engineers are mostly complicit because they get that 200k$ salary when they chase next shiny thing. No one is paying such salaries for mundane clerical job.
- chrisjj 2y ago
  You're saying complicity is not a fault?? :)
- binary132 2y ago
  Lead engineers and architects chasing resume-filler projects are absolutely part of the problem

msoad 2y ago

Kinda ironic that the article itself was updated

CooCooCaCha 2y ago

I disagree, keep things constantly updated (within reason). Most companies I've worked for have the attitude of the author, they treat updates as an evil that they're forced to do occasionally (for whatever reason) and, as a result, their updates are more painful than they need to be. It's a self-fulfilling prophecy.

eXpl0it3r 2y ago

Even as a developer not focused on web dev this sounds pretty bad, unless everyone in your dependency tree (from OS to language to libraries) decides to make a switch and even then, you'll be stuck with outdated ways to do things. Who wants to continue maintaining C++03 code bases without all the C++11/14/17/20 features? Who wants to continue using .NET Framework, when all the advances are made in .NET? Who wants to be stuck with libraries full of vulnerabilities and who accepts the risk? Not really addressed is the issue of developers switching jobs/projects every few years. Nobody is sticking around long enough to amass the knowledge needed to ensure maintenance of any larger code base. Which is caused by or caused the companies to also not commit themselves for any longer period of times. If the company expects people to leave within two years and doesn't put in the monetary and non-monetary effort to retain people, why should devs consider anything longer than the current sprint?

KronisLV 2y ago
> Who wants to continue maintaining C++03 code bases without all the C++11/14/17/20 features? Who wants to continue using .NET Framework, when all the advances are made in .NET? Who wants to be stuck with libraries full of vulnerabilities and who accepts the risk? With the exception that in this hypothetical world we'd get backported security updates (addressing that particular point), who'd want something like this would be the teams working on large codebases that: - need to keep working in the future and still need to be maintained - are too big or too time consuming to migrate to a newer tech stack (with breaking changes in the middle) with the available resources - are complex in of themselves, where adding new features could be a detriment (e.g. different code styles, more things to think about etc.) Realistically, that world probably doesn't exist and you'll be dragged kicking and screaming into the future, once your Spring version hits EOL (or worse yet, will work with unsupported old versions and watch the count of CVEs increase, hopefully very few will find themselves in this set of circumstances). Alternatively, you'll just go work somewhere else and it'll be someone else's problem, since there are plenty of places where you'll always try to keep things up to date as much as possible, so that the delta between any two versions of your dependencies will be manageable, as opposed to needing to do "the big rewrite" at some point. That said, enterprises already often opt for long EOL Linux distros like RHEL and there is a lot of software out there that is stuck on JDK 8 (just a very visible example) with no clear path of what to do once it reaches EOL, so it's not like issues around updates don't exist. Then again, not a lot of people out there need to think about these things, because the total lifetime of any given product, project, their tenure in the org or even the company itself might not be long enough for those issues to become that apparent.

pron 2y ago

Because of this, in the JDK we've adopted a model we call "tip & tail". The idea is that there are multiple release trains, but they're done in such a way that 1/ different release trains target different audiences and 2/ the model is cheap to maintain -- cheaper, in fact, than many others, especially that of a single release train. The idea is to realise that there are two different classes of consumers who want different things, and rather than try to find a compromise that would not fully satisfy either group (and turns out to be more expensive to boot), we offer multiple release trains for different people. One release train, called the tip, contains new features and performance enhancements in addition to bug fixes and security patches. Applications that are still evolving can benefit from new features and enhancements and have the resources to adopt them (by definition, or else they wouldn't be able to use the new features). Then there are multiple "tail" release trains aimed at applications that are not interested in new features because they don't evolve much anymore (they're "legacy"). These applications value stability over everything else, which is why only security patches and fixes to the most severe bugs are backported to them. This also makes maintaining them cheap, because security patches and major bugs are not common. We fork off a new tail release train from the tip every once in a while (currently, every 2 years). Some tail users may want to benefit from performance improvements and are willing to take the stability risk involved in having them backported, but they can obviously live without them because they have so far. If their absence were painful enough to justify increasing their resources, they could invest in migrating to a newer tail once. Nevertheless, we do offer a "tail with performance enhancements" release train in special circumstances (if there's sufficient demand) -- for pay. The challenge is getting people to understand this. Many want a particular enhancement they personally need backported, because they think that a "patch" with a significant enhancement is safer than a new feature release. They've yet to internalise that what matters isn't how a version is called (we don't use semantic versioning because we think it is unhelpful and necessarily misleading), but that there's an inherent tension between enhancements and stability. You can get more of one or the other, but not both.

deleted 2y ago
[deleted]
jldugger 2y ago
Is this any different than the LTS approach Canonical and others take?

ricksunny 2y ago

The 'Skip this Update [pro]' button example (Docker Desktop) just made me facepalm and helped me internalize that I'm not a luddite from technology, I'm a luddite from the collectives of people (not the individual people...(!) ) feeling compelled to craft these dark business patterns.

dzonga 2y ago

java over golang lol. when golang has literally been version stable for over a decade now

canpan 2y ago
I think the article is interesting. But you are right, golang is the one language I would recommend to someone who does not like updates.

cupantae 2y ago

I’ve supported enterprise software for various big companies and I can tell you that most decision makers for DCs agree with this sentiment. EMC had a system called Target Code which was typically the last patch in the second-last family. But only after it had been in use for some months and/or percentage of customer install base. It was common sense and customers loved it. You don’t want your storage to go down for unexpected changes. Dell tried to change that to “latest is target” and customers weren’t convinced. Account managers sheepishly carried on an imitation of the old better system. Somehow from a PR point of view, it’s easier to cause new problems than let the known ones occur.

_yb2s 2y ago

I pretty much agree- most systems don't need updating. I've seen and setup OpenBSD servers that ran for a decade without issues never getting updates. I currently run some production web services on Debian where I do updates every 3 years or so, and no issues. Leaving something alone that works good is a good strategy. Most of the cars on the road are controlled by ECUs that have never had, and never will have any type of updates, and that is a good thing. Vehicles that can get remote updates like Teslas are going to be much less reliable than one not connected to anything that has a single extensively tested final version. An OS that is fundamentally secure by design, and then locked down to not do anything non-essential, doesn't really need updates unless, e.g. it is a public facing web server, and the open public facing service/port has a known remote vulnerability, which is pretty rare.

hcarvalhoalves 2y ago
> Vehicles that can get remote updates like Teslas are going to be much less reliable than one not connected to anything that has a single extensively tested final version. I don’t think it’s necessarily this, but the fact that being able to update anytime is a great source of pressure to release untested software at any cost.

albertP 2y ago

He. I've been running always everything one or two versions behind latest (for my personal laptop, not servers). That means mainly OS (e.g., macOS), but as long as I can avoid automatic updates, I do so. I believe the chances of having a bricked laptop because of a bad update are higher than the chances of getting malware because running one or two versions behind the latest one.

Plasmoid 2y ago

I was working at a place that delivered onprem software. One customer asked us "We like features of version N but we're running N-1. Can you backport them so you don't have to upgrade?". I replied we'd already done that, it was called version N.

yjftsjthsd-h 2y ago
I mean... there could be a case for "does this feature really require breaking changes?" that would distinguish those.

mike741 2y ago

Urgent updates can be necessary every once in a while but should be recognized as technical failures on the part of the developers. Failure can be forgiven, but only so many times. The comments saying "what about X update that had this feature I need?" are missing the point entirely. Instead ask yourself about all of the updates you've made without even looking at the patch notes, because there are just too many updates and not enough time. Instead of blaming the producers for creating a blackbox relationship with the consumers, we blame the consumer and blindly tell them to "just update." That's what needs to change. It's a bit similar to opaque ToS issues.

ghawk1ns 2y ago

There is humor in this blog itself has 2 updates

userbinator 2y ago

There is also another type of update: security updates that don't actually matter in the environment that the software is used in. The question of whether the "new features" are for or against the user is another point to ponder.

cpncrunch 2y ago

A reasonable strategy is to wait a week after release before applying an update, unless it's a zero day fix.

palata 2y ago

> My experience shows that oftentimes these new and supposedly backwards compatible features still break old functionality. Well that's the first issue: downright malpractice. Developers should learn how to know (and test) whether it is a major change or not. The current situation is that developers mostly go "YOLO" with semantic versioning and then complain that it doesn't work. Of course it doesn't work if we do it wrong.

palata 2y ago

What the article points to is that most updates are bad updates. We teach people that they should accept all updates for security reasons, but really they should only accept security updates. But they can't, because this is not a possibility that is given to them. All updates are put together, and we as an industry suck at even knowing if our change is backward compatible or not (which is actually some kind of incompetence). And of course it's hard, because users are not competent enough to distinguish good software from bad software, so they follow what the marketing tells them. Meaning that even if you made good software with fewer shiny features but actual stability, users would go for the worse software of the competitor, because it has the latest damn AI buzzword. Sometimes I feel like software is pretty much doomed: it won't get any better. But one thing I try to teach people is this: do not add software to things that work, EVER. You don't want a connected fridge, a connected light bulb or a connected vacuum-cleaner-camera-robot. You don't need it; it's completely superfluous. Also for things that actually matter, many times you don't want them either. Electronic voting is an example I have in my mind: it's much easier to hack a computer from anywhere in the world than to hack millions of pieces of paper.

hulitu 2y ago
> And of course it's hard, because users are not competent enough to distinguish good software from bad software, There was a time when Windows had a description for updates. Now the only distinction is between KB3587690 and KB67457770.
o11c 2y ago
I also noticed that there's another kind of update not mentioned: * Updates that add a theoretically independent feature, but which other software will dynamically detect and change their behavior for, so that it's not actually independent.
ranger207 2y ago
And you can't have _just_ security updates, because the combinational complexity of security fixes across feature versions is insane, not to mention the fact that the interaction of the security and features changes can themselves introduce bugs. There's groups that try (eg distro maintainers) but it's ultimately a losing battle. I'm convinced that patching is only a bandaid, but it's also impossible to have 100% bug-free code, so there needs to be some sort of systematic solution on top of whatever particular code is running. Behavior analysis, egress network analysis, immutable by default images with strictly defined writeable volumes and strictly defined data that's going to be written there, etc. There's not a silver bullet, but I think patching and trying to keep up with updates is, like, a gallium bullet at best

tonymet 2y ago

our industry could use a risk assessment index scanner on updates, similar to "npm audit" , that measures the delta between versions and gives a risk indicator based on a number of parameters. The issue with changelogs is that they are an honor system, and they don't objectively assess the risk of the update. Comparing changes in the symbol table and binary size could give a reasonable red/yellow/green indicator of the risk of an update. Over time, you could train a classifier to give even more granularity and confidence.

k_roy 2y ago

> In my eyes it could be pretty nice to have a framework version that's supported for 10-20 years and is so stable that it can be used with little to no changes for the entire expected lifetime of a system. Yeah, me too. I also would like a few million bucks in the bank. It's naive to think that every project wouldn't want to set this goal, simply because it's so unrealistic.

walterbell 2y ago

> And lastly, choose boring technology. Only use software that you're sure will be supported in 10 years. Only use software, which has very few "unknown unknowns". Only use software, where the development pace has slowed down to a reasonable degree. Perl has been stable for a couple of decades.

iso8859-1 2y ago
Perl 6 is incompatible in all sorts of ways. It's so incompatible that people didn't even upgrade! So incompatible that they renamed the language! Even more incompatible than Python 3! Even Perl 5 is rapidly evolving. They just added try/catch. Added a new isa operator. Added a new __CLASS__ keyword. Added defer blocks.

binary132 2y ago

I had started to think I was the only one saying this.

gtirloni 2y ago

The world is not static and software these days is very interconnected. Dreams of not updating only work in a unchanging world. Sadly, this world is still to be found.

LargoLasskhyfv 2y ago
Funny you're saying that. In context of the stricken crowds I've read that FedEx, UPS and SouthWest had no problems at all, because many systems run on Windows95, or even something based on Win3.1x :-) Seems they are well isolated, but networked nonetheless. Or do you think they use bush-drums and smoke signs to conduct their businesses?
- Groxx 2y ago
  That probably has more to do with not jumping on the latest fad* and installing an unattended unrestricted automatically-globally-updating rootkit on all of your critical machines. That is generally recognizable as stupidity. And the ones that did so are now paying the price. * compliance tactics are very prone to fads. just look at cookie banners.

hcarvalhoalves 2y ago

Extreme viewpoint, but agree strongly. Big reason why working in Common Lisp brings a smile to my face - it’s a standard, quicklisp works, ffi works, etc. I can run code and follow instructions written DECADES ago, it just damn works.

1vuio0pswjnm7 2y ago

Avoid software that will need constant updates. Because that is a signal it is defective to begin with, or expected to be broken soon. For example, I avoid graphical commercial OS, large, graphical web browsers. Especially mobile OS and "apps". Avoidance does not have to be 100% to be useful. If it defeats reliance on such software then it pays for itself, so to speak. IMHO. The notion of allowing RCE/OTA for "updates" might allegedly be motivated by the best of intentions. But these companies are not known for their honesty. Nor for benevolence. And let's be honest, allowing remote access to some company will not be utilised 100% for the computer owner's benefit. For the companies remotely installing and automatically running code on other peoples' computer, surveillance has commercial value. Allowing remote access makes surveillance easier. A cake walk.

m3kw9 2y ago
Exception is mobile apps

Groxx 2y ago

To jump on a related article since it's linked and comments are now closed: https://blog.kronis.dev/articles/stable-software-release-system https://blog.kronis.dev/articles/stable-software-release-sys... The 2021/2022/2023/2024 version-numbering schemes are for applications, not libraries, because applications are essentially not ever semver-stable. That's perfectly reasonable for them. They don't need semver. People don't build against jetbrains-2024.1, they just update their stuff when JetBrains breaks something they use (which can happen at literally any time, just ask plugin devs)... because they're fundamentally unstable products and they don't care about actual stability, they just do an okay job and call it Done™ and developers on their APIs are forced to deal with it. Users don't care 99%+ of the time because the UI doesn't change and that is honestly good enough in nearly all cases. That isn't following semver, which is why they don't follow semver. Which is fine (because they control their ecosystem with an iron fist). It's a completely different relationship with people looking at that number. For applications, I totally agree. Year-number your releases, it's much more useful for your customers (end-users) who care about if their habits are going to be interrupted and possibly how old it is. But don't do it with libraries, it has next to nothing to do with library customers (developers) who are looking for mechanical stability.

kkfx 2y ago

Or use NixOS/Guix Systems instead of living in the stone age of containers...

k_roy 2y ago
Or, alternatively, realize both are valid options and have different use-cases. Or big-brain it and use Nix to build your containers and get the best of both worlds.
notfed 2y ago
I ran into critical issues with some NixOS packages I needed, the maintainer doesn't seem to care about my use case, and I can't for the life of me understand the Nix language enough to fix it myself. Have stopped using NixOS because of that. If Nix language was replaced with something sensible I'd jump back in excitedly.
- kkfx 2y ago
  The point is not much the project per se but the concept, Infrastructure as Code, OS as code. So far NixOS is a relatively small ecosystem, in a complex community situation with the main sponsors, a mil-tech company, who have got a wave of backslash for debatable reasons, Guix System on other side it's mostly a French INRIA project focused more on HPC than anything else, so both have their corners, but they show a modern way to develop and deploy OSes as OpenSolaris (since it was before IllumOS fork) show the first integration of storage (zfs) and package system (IPS) and bootloader (with the boot environment concept), somewhat badly copied later by FreeBSD and partially on GNU/Linux with a set of scripts. ALL OTHER OSes/distro are still stuck in the '80s in that sense and most people seems even unable to understand. On storage alone the famous "rampant layer violation" and the absurdity of btrfs and stratis "against" zfs are really good examples of blind tech reactionary behaviors by high skilled people and their outcomes a showcase of why we damn need to innovate instead of shooting ourselves in the feet switching from something obsolete to something even worse (like the now-almost-finished full stack virtualization on x86 and thereafter the paravirtualization/container mania still current) layering crap on crap with more and more unmanageable infra with enormous attack surfaces. Another small and relatively known example: Home Assistant project: apart of their design, they choose to distribute a python application as a GNU/Linux entire distro because to the such move seems to be commercially sound and many others choose to follow them instead of simply pip-install HA in a local venv, wasting an immense amount of resources on their system for what? Such kind of tech evolution must end or we will collapse soon digitally speaking.

aspyct 2y ago

There's an alternative solution: update everything, but limit your dependencies. Example: for my (personal) projects, I only use whatever is available in the debian repositories. If it's not in there, it's not on my dependency list. Then enable unattended upgrades, and forget about all that mess.

mikewarot 2y ago

When I was the sole IT guy for a small consulting company, once I got everything working, I never updated it. We used Microsoft office 2000 for 12 years. Never had to retrain people, deal with the weird ribbon toolbar, etc. It's only the deranged use of OSs with ambient authority that gums up what would otherwise be stable systems.

msiemens 2y ago

I know it's this is a rather long tangent and not the main point of the article, but regarding "Docker Swarm over Kubernetes", I've had a ton of bad experiences at my employer running a production Swarm cluster. Among them: - Docker Swarm and Docker Compose use different parsers for `docker-compose.yaml` files, which may lead to the same file working with Compose but not with Swarm ([1]). - A Docker network only supports up to 128 joined containers (at least when using Swarm). This is due to the default address space for a Docker network using a /24 network (which the documentation only mentions in passing). But, Docker Swarm may not always show error message indicating that it's a network problem. Sometimes services would just stay in "New" state forever without any indictation what's wrong (see e.g. [2]). - When looking a up a service name, Docker Swarm will use the IP from the first network (sorted lexically) where the service name exists. In a multi-tenant setup, where a lot of services are connected to an ingress network (i.e. Taefik), this may lead to a service connecting to a container from a different network than expected. The only solution is to always append the network name to the service name (e.g. service.customer-network; see [3]). - Due to some reason I still wasn't able to figure out, the cluster will sometimes just break. The leader loses its connection to the other manager nodes, which in turn do NOT elect a new leader. The only solution is to force-recreate the whole cluster and then redeploy all workloads (see [4]). Sure, our use case is somewhat special (running a cluster used by a lot of tenants), and we were able to find workarounds (some more dirty than others) to most of our issues with Docker Swarm. But what annoys me is that for almost all of the issues we had, there was a GitHub ticket that didn't get any official response for years. And in many cases, the reporters just give up waiting and migrate to K8s out of despair or frustration. Just a few quotes from the linked issues: > We, too, started out with Docker Swarm and quickly saw all our production clusters crashing every few days because of this bug. […] This was well over two years (!) ago. This was when I made the hard decision to migrate to K3s. We never looked back. > We recently entirely gave up on Docker Swarm. Our new cluster runs on Kubernetes, and we've written scripts and templates for ourselves to reduce the network-stack management complexities to a manageable level for us. […] In our opinion, Docker Swarm is not a production-ready containerization environment and never will be. […] Years of waiting and hoping have proved fruitless, and we finally had to go to something reliable (albeit harder to deal with). > IMO, Docker Swarm is just not ready for prime-time as an enterprise-grade cluster/container approach. The fact that it is possible to trivially (through no apparent fault of your own) have your management cluster suddenly go brainless is an outrage. And "fixing" the problem by recreating your management cluster is NOT a FIX! It's a forced recreation of your entire enterprise almost from scratch. This should never need to happen. But if you run Docker Swarm long enough, it WILL happen to you. And you WILL plunge into a Hell the scope of which is precisely defined by the size and scope of your containerization empire. In our case, this was half a night in Hell. […] This event was the last straw for us. Moving to Kubernetes. Good luck to you hardy souls staying on Docker Swarm! Sorry, if this seems like like Docker Swarm bashing. K8s has it's own issues, for sure! But at least there is a big community to turn to for help, if things to sideways. [1]: https://github.com/docker/cli/issues/2527 https://github.com/docker/cli/issues/2527 [2]: https://github.com/moby/moby/issues/37338 https://github.com/moby/moby/issues/37338 [3]: https://github.com/docker/compose/issues/8561#issuecomment-1873501393 https://github.com/docker/compose/issues/8561#issuecomment-1... [4]: https://github.com/moby/moby/issues/34384 https://github.com/moby/moby/issues/34384