25 ms·

Long term support considered harmful

207 points by luu 12y ago

jeffdavis 12y ago
PostgreSQL has a 5-year support window. I'm curious what negative consequences the author thinks PostgreSQL suffers as a result from that policy.
- silvestrov 12y ago
  It's a lot easier with a 5 year window for PostgreSQL: Kernel developers have to support all kinds of weird semi-buggy hardware that they can't buy in their country, they have to make the kernel work with badly written GPU drivers , and they have to deal with interrupt-level code which is notoriously difficult to debug.
  - jeffdavis 12y ago
    The author's primary example for the "LTS is bad claim" was glibc, which is not a kernel. Are you saying the "LTS is bad claim" only applies to kernels and other things very close to the hardware (closer than a database)?
fidotron 12y ago
Upvoted for some interesting points, but it's quite wrong for the simple reason that on a commercial timescale one year is nothing. There are reasons Microsoft end up being dragged into supporting old OS versions, and it's because running an operating system is just a means to an end, which hopefully is doing something useful. "Upgrades" have, understandably, become synonymous with unnecessary pain and breaking things. Maybe OpenBSD really is more rigorous about quality control, but (as an example) if you were to just accept every Ubuntu update it wants to install you'd be wasting a significant amount of time just ensuring your system works properly.
- deleted 12y ago
  [deleted]
  - specialp 12y ago
    This is implying that people actually want to upgrade full releases. Perhaps I would like to make a Linux box and have it last a few years, and only patch it for security fixes as it is working fine. Yes if I do have to upgrade that system 5 years later it is going to be more painful than every 6 months, but if I did it every 6 months I would have to devote development time and switch context every 6 months. Small updates are better if you have to upgrade entire systems, but many people do not want to risk destabilizing working systems every 6 months. For that we have LTS where someone will integrate in security fixes in a hopefully non impacting fashion and for that I am thankful
- deathanatos 12y ago
  > on a commercial timescale one year is nothing. I've worked in commercial environments where common invocations of `tar` did not work, because `tar` was a decade out of date. I had to learn things and habits that had died before I even started programming. It wasn't pleasant. Do not underestimate the age and stubbornness to upgrade of some environments. Recently, I've helped migrate software from Ubuntu Precise to Trusty, and the amount of differences make things mildly frustrating. We don't get to just run one or the other, and we can't just drop everything and move to Trusty. We have to continue to support the old while we build support for the new, briefly support both, transition to the new, then tear out the support for the old. Migrated. It's a lot of work when the changes are huge, but much more manageable when I can take the changes more piecemeal (it's one if statement, as opposed to many, that I need to manage at any given time). That's in a production environment. I run Gentoo at home. I much prefer its rolling releases to Ubuntu and Debian which I ran alongside and before, respectively. Things break every now and then. It's a tad annoying, but it gets fixed sooner or later.
zokier 12y ago
Isn't 6 month release cycle and support for past 2 releases essentially also the Fedora release model? From what I've heard (and used) that hasn't been massively successful. > No one upgrade is likely to end in disaster because there simply isn’t enough change for that to happen. That might be true for OpenBSD, but in Linux land the rate of change certainly is great enough to cause major breakage even at 6 month cycle.
craigds 12y ago
"Considered Harmful" Essays Considered Harmful: http://meyerweb.com/eric/comment/chech.html http://meyerweb.com/eric/comment/chech.html I think the OP is being a bit naïve if they expect all users to upgrade to a new OS every year. Upgrades of Ubuntu and OSX are usually quite painful endeavours, fraught with UI-breakage and new bugs to solve, and there's usually little incentive for users to upgrade in every 6-monthly release. IMHO Ubuntu has a good two-pronged approach - short-term support for most releases, but a LTS every couple of years for those who don't want to handle the pain of upgrading all the time. Anecdotally, even upgrading to stay up-to-date with the LTS versions can be difficult. A company I deal with is currently scrambling to ditch Ubuntu 10.04 before it loses support in April this year. That's 5 years old now. Companies don't upgrade for fun.
chollida1 12y ago
> A one year support window isn’t too short; it’s too long. Wow, I'm in very strong disagreement with the author on this point. I don't think I could find a single person, technical or otherwise who didn't use the same operating system for more than a year. In fact its the default for almost everyone who isn't technically inclined to upgrade their os only when they get a new computer. What about every single car produced today. I doubt you'd want to upgrade them every 6 months. As a ubuntu user, I find that I cross my fingers every time I run sudo apt-get upgrade. About half the time I get broken builds and once a year my OS will just flat out crash from it, or fail to reboot in virtual box. This view, while idealistic is just so laughable I can't begin to believe the author is serious. I've worked with some mission critical systems( stock exchange's) They take more than a year planning a big OS upgrade. To turn around and tell them that they should be doing this every 6 months is just so out of touch with reality, I don't know how to respond to the author:( Many companies won't install a new piece of software until its been proved out in production for 6 months to a year by someone else....
- deleted 12y ago
  [deleted]
  - zokier 12y ago
    > That's what you need to do when OS releases are made only once every few years and are big. They don't have to be that way Ubuntu and Fedora releases happen already in 6 month cycles. This is not some novel unexplored concept.
    - wtallis 12y ago
      Ubuntu and Fedora are aggressive about incorporating new stuff. It's possible to have the same release cadence while operating with a more Debian-like attitude toward stability.
  - taeric 12y ago
    I question the scale of just what is getting upgraded if it is "quick and painless." Note, that I do not mean to diminish the accomplishment. Especially as a fan of TeX and someone that often toutes that maybe we are learning the wrong software lessons by constantly rewriting everything to be in the latest safe language/paradigm. However, to not acknowledge all of the progress that is made in other areas is just as dangerous as not acknowledging the benefits of more deliberate methodologies.
zdw 12y ago
The downside of this is that OpenBSD is pretty difficult to upgrade in a clean manner given it's "uncompress the tarball over /, then clean stuff up manually" method. I really like OpenBSD, but it desperately needs a modern package management system. This is frankly my largest sticking point with it - it's faster and much cleaner in most cases to wipe, reinstall, then apply configuration than to try to upgrade.
- deleted 12y ago
  [deleted]
- jkot 12y ago
  How about debian with freebsd kernel?
  - elektronjunge 12y ago
    The new pkg tool in freebsd is pretty good and meshes nicely with the ports tree. The marginal improvements for 3rd-party built packages with apt aren't really worth getting rid of the rest of freebsd for. It also doesn't help OpenBSD, which really needs a new package manager.
- danielweber 12y ago
  As someone who used to help maintain hundreds of OpenBSD boxes, yes. But don't forget, OpenBSD is designed for the developers.
- brynet 12y ago
  This is oft-repeated, but how is OpenBSD's package management system "not modern"? It handles upgrades properly, modified configuration in /etc is preserved. Packages are cryptographically signed as of 5.5, files are individually checksummed.. I'd say OpenBSD still handles dependencies and upgrades better than FreeBSD's supposedly "next-generation" pkg. OpenBSD's package system and ports tree are tightly integrated and work together, and despite the historic naming, OpenBSD's pkg_{add,delete,info,create}(1) are unrelated to the now deprecated FreeBSD equivalents.
- 12y ago
IshKebab 12y ago
Can we have a ban on "X considered harmful" titles? Have some originality.
jerf 12y ago
In theory, I agree. I've often likened code development to muscles; you get good at what you do, and what you don't do atrophies. If you push changes ever hour, you get very good at making sure that works. If you push them every three months, you get good at pushing them every three months, and panic when something requires you to move within days. And so on. This applies to all sorts of development process aspects (test a lot, get good at testing, fail to test, becomes impossible to test later, etc). In practice, neither what tedunangst nor I think matters, because once you ship something to a customer in any way, they're not going to upgrade it. Offer automated upgrades and they'll demand a way to turn them off. It's not just developer boxes we have to worry about, unfortunately.
gtirloni 12y ago
Oh please, all it would take for this "Ghost" vulnerability not to happen was someone at glibc to have made the right call regarding how to treat a buffer overflow in a function dealing with external data. They didn't, so what? The lesson here is that Red Hat & other companies should do additional review if they are shipping something branded as super secure and stable. Long term support is great. People screw up sometimes. Additionally, it's about time core software components get more attention from companies with the big bucks (and that are profiting from it).
- vezzy-fnord 12y ago
  Personally I'm of the idealistic belief that glibc is really starting to show its corrosion, and should be replaced by a clean-room, standards-compliant and elegant design like musl: http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc http://wiki.musl-libc.org/wiki/Functional_differences_from_g... Of course, this would be a massive change for the Linux ecosystem at large. It is my hope that projects like Sabotage Linux will eventually procure most major software to become musl-compatible, though. The source of the GHOST bug was in NSS, which musl lacks by design.
  - GregBuchholz 12y ago
    Speaking of corrosion, maybe we should be building systems with a language which avoids most of the problems of C.
    - Silhouette 12y ago
      The trouble with that otherwise perfectly reasonable position is that no such language exists yet that doesn't have its own significant problems of one kind or another. I'm hopeful that this situation will finally change within the next decade and that C and C++ can be honourably discharged as soon as possible thereafter. However, I don't see this happening until newer languages with sounder foundations -- Rust comes to mind as a promising example -- have entered the mainstream. Until then, the ecosystem around C and C++ has too much momentum for a lot of projects to switch to more experimental technologies even if those technologies have advantages in terms of robustness.
zokier 12y ago
Personally I think a 6 month release cycle is the worst of both worlds. Either give me LTS and I'll do that one big upgrade every few years, or give me rolling release where any breakage should be relatively localized and minor. But with 6 month release cycle you have to constantly be upgrading, but the upgrades are still big "break the world" upgrades.
allendoerfer 12y ago
His reasoning makes sense but, as all developers tend to do, he spots an error in a system and wants to switch to another or invent a new one. You know the xkcd. Often times the solution is just to work together and fix the old one. For me the logical fix would be to patch all kinds of malicious, undefined or non-spec behaviors in LTS releases in short cycles regardless whether the developer thinks, it is security-critical or not. To make this more feasible you could either pay for it or use a minimal base system separated from the user-space. Both exists today.
- JoachimSchipper 12y ago
  That's both a lot of work and a lot of churn, though - people choose LTS because they don't want things to change!
  - allendoerfer 12y ago
    In a sense, things do not change, they stop to be changed from a assumed state.
Sanddancer 12y ago
There are a lot of problems with this argument, especially when you go into the ports section. For example, the version of go between 5.4, 5.5, and 5.6 is different for all three versions, which means you're going to have to rebuild packages, run regression tests, etc every six months to make sure some new feature doesn't run into your code, or some bug isn't introduced. Similarly, in the case of the base install, you are still going to have to run a regression test there to ensure that any deprecated or removed features don't impact your workflow. These things are expensive from a development and ops perspective, and why most commercial software vendors tend to stick to a releasing for a few platforms they know are going to be supported for longer than a year or two. Yes, OpenBSD upgrades are mostly painless, but still, bugs can and do happen, and expecting everyone to be able to just drop everything and rebuild is plain and simple unreasonable.
learnstats2 12y ago
>Now on the one hand, this forces users to upgrade at least once per year. The problem with this article is this assumption which is not true in the slightest. There's no force which makes users upgrade - some users will upgrade when they're told; some will wait 5 or more years - and may have good reason to. > Nothing kills a bug report faster than “My network card worked in 4.4 but stopped working in 5.6.” Developers aren’t going to bisect five years of changes; you get to do that yourself. This statement can translate into don't ever upgrade. Is that really the goal?
- sarciszewski 12y ago
  No. Ideally, the goal is that users will upgrade as soon as a fix is available.
  - taeric 12y ago
    I would think ideally, programmers wouldn't break things that are working. We know that ideal doesn't pan out.
    - sarciszewski 12y ago
      "programmers wouldn't break things that are working" So you want the world's best security experts to manually review every module of every program you use, as well as every piece of every operating system you want it to run on, to make sure there are 100% certainly no exploitable bugs that could require a fix down the line? That's the same argument, but with reality applied. Programmers don't "break things that are working", people find design flaws that could lead to remote exploitation. If you don't patch your stuff, enjoy being 0wned, not my problem. For the rest of the world, bleeding edge is the safest place to be.
      - taeric 12y ago
        Yes, yes I would love that. Though, as I noted, nobody expects that ideal to pan out. Reality applied to many large organizations to the prospect that things may break on an upgrade translates to "don't upgrade." And really, this isn't necessarily bad. So long as what isn't getting upgraded has been compartmentalized and does its job correctly. Consider, we don't even think about asking companies to upgrade processors on a yearly basis. Monitors? Good for years. Actually, pretty much all of the hardware. Why do we think software should be different?
jacquesm 12y ago
There is a cost to upgrading and there is a cost to staying behind. As soon as the cost to stay behind outweighs the cost to upgrade I upgrade, but no sooner. Otherwise I'd just be throwing away money and time. The bigger problem here is that the cost to upgrade is only known after the fact. What should be a routine job can easily spiral out of control into a marathon of misery.
thaumaturgy 12y ago
> Frequent upgrades amortize the cost and ensure that regressions are caught early. No one upgrade is likely to end in disaster because there simply isn’t enough change for that to happen. Oh, how I wish this were true. For what it's worth, it's pretty true as far as OpenBSD is concerned, in my experience. But OpenBSD is the exception here, not the rule. Everywhere else, developers all seem to have embraced "break early, break often". Eventually you get burned. For me, it was a routine should-have-been-minor web server update where one of the packages I relied on suddenly became unsupported and every single hosted site stopped working. Since there's no way to roll back server upgrades, I had a marathon night involving building a new server stack and migrating all hosted sites there by 8 a.m. But you can't yell at anybody when that happens, because the answer's always the same: it's not the developers' fault. Who really believes sysadmins wouldn't update everything all the time if they could? Old, dodgy, out-of-date servers exist exactly because updates are butthole-puckering, because everyone's been burned at least once by a "minor" update, and because once the damage is done, undoing it is horrifyingly difficult.
- edofic 12y ago
  "Since there's no way to roll back server upgrades" It is if you run a modern filesystem like ZFS or btrfs. You just do a cheap snapshot before upgrading(can be automated) and roll back if there are problems. Even works with lvm.
  - thaumaturgy 12y ago
    Ooh, thanks. That's a good idea. I'd never considered running ZFS on my small servers.
    - keithpeter 12y ago
      Or perhaps simply gzip of the / partition before upgrading. If insuperable problems, zap new / and restore old known working one? Assuming data/sites are on different partitions.
      - 12y ago
Silhouette 12y ago
This is not an argument against long term support. It is an argument against using OpenBSD for any project you need to work for more than one year.
CrLf 12y ago
Forcing users to upgrade frequently has one outcome, and one outcome only: they will stop upgrading, period. If you give users only two choices, they will choose one, but the one that requires less effort. There is more to life than keeping up with the constant upgrade treadmill. It's already bad as it is. Some Linux distributions are on a 3-year support cycle, which is too short. If you have many servers, especially many servers running a lot of different things, upgrading every 3-years means having people doing nothing more than upgrades. Once you fix your applications to support the changes in the next OS release, that release is almost out of support...
- wmf 12y ago
  they will stop upgrading, period Has that been observed with Chrome, ChromeOS, iOS, etc.?
  - CrLf 12y ago
    That's an apples to oranges comparison. To upgrade Chrome a consumer depends only on himself. To upgrade a server OS, the "user" (the sysadmin) depends on a number of other people with different priorities. And that's _if_ the people he depends on are still with the company... I've been faced inumerous times with impossible upgrades (nobody cares about upgrading besides you, so nobody cares to fix whatever is preventing the upgrade). And when the upgrade is possible, many times it takes years. That same user that upgrades Chrome on his laptop, is still stuck with IE7 at work because his company relies on it to pay his salary.
slasaus 12y ago
It's funny that just today, before I found out about the glibc vulnerability, I reconsidered if I really want to upgrade my Ubuntu 10.04 mail and web servers to Ubuntu 14.04 LTS. I was triggered after reading some bad things about 14.04 [1]. I've looked at the M:Tier binpatches and package upgrades for OpenBSD, looked at FreeBSD, Debian with it's experimental LTS, but eventually I'm still in favor of the 5-year support for Ubuntu server. I have bad experiences in upgrading production machines in-place, whether it is OpenBSD, Ubuntu or Debian and always install a new machine (vps) side-by-side which is really the only stress-free guaranteed way to go in my opinion. Having to do this only once every 5 years is really a lot nicer then at least once a year. The good security backports of Ubuntu, minimal breakage (auto-security upgrades at my Ubuntu servers have been working almost flawlessly [2]) are the least maintenance, stable and secure setup I can imagine. OpenBSD having only one year of support, no binpatches of the kernel and having no stable security fixes of the packages are the reason I only use it with anything that can be done by the base system (backup host and nameserver). OpenSMTPd looks very promising, but I would need supported amavisd packages, same goes for httpd that needs php in my setup. Besides it's limited use, I still love OpenBSD and the mindset that stewards it. If only they had longer support and binpatches for kernel and packages :) [1] https://tim.siosm.fr/blog/2014/04/25/why-not-ubuntu-14.04-lts/ https://tim.siosm.fr/blog/2014/04/25/why-not-ubuntu-14.04-lt... [2] the mail config was overwritten once after an auto security update of dovecot in 10.04, quickly recovered it with etckeeper (/etc in git)
- tghw 12y ago
  Thanks for the tip on etckeeper! I've been wanting something like that for a while.
coherentpony 12y ago
"[Useful popular thing] considered harmful."
jrochkind1 12y ago
The only thing that would make this realistic is if open source developers prioritize backwards compatibility very highly. Of course, there will always be some bugs anyway. And it varies across language (or other open source affinity) communities. But in many communities, developers aren't really even _trying_ to ensure backwards compatibility. If developers committed to backwards compatibiilty with successive versions for X number of years -- then you could update to the latest release for a number of years worrying less about breaking things, rather than needing to freeze the versions you use for that number of years. Yes, this would significantly increase developer hours on open source projects. Nothing comes for free. But wouldn't those developer hours be better spent centrally on ensuring backwards compatibility in the projects themselves; as opposed to every developer of every consumer of the project needing to deal with backwards incompatibility on every upgrade? Or every distro needing to backport patches? (This latter is more debatable, which is exactly why we have the status quo)
kbenson 12y ago
Every time this topic comes up, we have people talking about really different things not clearly explaining what they are referring to, and causing confusion and people not understanding the reasoning of others. I've read the following arguments already, but with portions of them implicit. See if you can determine why the people are talking at cross points: - "I generate VM server images for all my needs, and deploy to virtualization infrastructure behind load balancers to handle services. I treat he OS like an application like the rest of the stack, and all my data is abstracted to a data layer. I just generate a new image with patches and test it, then deploy if it works in my test suite." - "I have hundreds of servers in dozens of roles with different software needs, and I need them to be secure and stable in a timely fashion, and I can't spend multiple weeks achieving that. Long term support and back-patching allows me the time to plan needed large changes in infrastructure without having to spend all my time managing updates, software changes and configuration changes multiple steps down the dependency graph." - "I have a few servers with a few roles, or many servers with one or two roles, and I can manage frequent updates just fine, and it allows me to take advantage of the newest features, get security updates immediately as the software authors fix it, and I don't have to worry about end of life of software." - "I have an application stack with multiple dependencies, and I just make sure to update my stack as I make changes to the software. I would love/have set up continuous integration software to build and test everything as I go, so I know if it works or not before taking it live." As someone who's been in all of these situations at different points, often multiple at once, let me be clear: Unless you have an argument that addresses ALL of these situations, you really haven't thought through the issue.
- hawleyal 12y ago
  I think the author is advocating that those scenarios should converge. His argument is that second scenario is an anti-pattern and should be corrected to be more inline. It might be a pipe dream, but it's still his perspective.
  - kbenson 12y ago
    They serve different needs. The second scenario may well be better served by moving towards the first, but that's not a quick project, especially when uptime and reliabiity is important. Even with that, the for the third scenario a full VM infrastructure may by overkill. This whole topic can be summed up with "Examine your needs, examine your options, make the best choice you can at the time, try to make your life easier by leaving pathways to change your choice with the least problems based on expected future needs." Just because someone looks to be doing something similar to you, doesn't mean their needs and constraints aren't quite different. I think this is the big thing most people miss.
contingencies 12y ago
Reading the title I assumed this was a rant about how proper ops processes replaced the need for LTS releases and the demonstrably bad practices they could be seen to encourage. IMHO, this would have been a more interesting and valid point. The current post is a straw man argument for a few reasons. First, because vulnerabilities will be exploited whether the window of availability is 1 day, 100 days, or 1000 days (like this one) and likewise, upgrades will be missed, for various reasons, whether or not they are encouraged. Second, the author spuriously implies that a 6 month sliding upgrade window ("one size fits all") approach is superior, easier and more appropriate for everyone through a combination of broad assumption and vague hand-waving about developer time efficiency. Rubbish. If you want newer code, then ASAP is the go and BSD-style monolithic releases are bad... something closer to the versionless OS ideal of constant, incremental, package-wise release processes such as Gentoo's portage (BSD-inspired!) where you can even install a -9999 (git HEAD) version of many packages would be ideal. Obviously, with such an approach, stability caveats must be considered (as they are). Thus, both FreeBSD and OpenBSD actually represent a 'middle-ground' between Gentoo and commercial Linux vendor LTS, with OS wide release processes incorporating some greater collective package and package interoperability testing, combined with the package-wise release of things through the ports trees. What's the real ideal here? A general purpose means to build working, tested, maintainable, secure systems with minimal effort for a broad audience. How to get there? Clearly, not by whinging about external release window frequencies of volunteer-based projects, as any frequency still creates bugs, and no process is perfect. The answer, I believe, is time-honored. Careful design for failure and minimalism, and good process. Test. Measure. Remove surplus features and components. Iterate.
ak217 12y ago
Spoken like someone who has never had to maintain a large production system. This notion is handily disproved by the market, though. Red Hat and Canonical enjoy commercial success and mindshare precisely because they provide stable long-term support platforms on which others can build software. Most of Canonical's worth is actually embodied in their commitment to LTS releases, which provide a sweet spot between stability and the glacial pace of RHEL.
reality_czech 12y ago
Yes, we should all step forward into the wonderful world of running brand new software all the time in production. I'm sure it will all be much more secure than relying on year old software that is being actively maintained. Is the date on this one supposed to be April 1st?
cesarb 12y ago
A problem with the lack of long term support is that you have to keep moving. The most important part of "long term support" distributions is that breaking changes are kept to a minimum. If a "long term support" distribution releases an update to for instance glibc, you can expect that applying that update will change nothing other parts of your software stack might depend on. The dependencies might be subtle; for instance, a new version of a database server might have optimized its query planner, which happened to make one particular query your software does a couple of percent slower, which led to it using a few more seconds to do its processing, enough to push it over the timeout limit for a different part of the system. So the only sane way to avoid breaking changes is to avoid all changes. The opposite would be, as advocated in this post, frequent upgrades. "Frequent upgrades amortize the cost and ensure that regressions are caught early", but that means you are dealing with upgrades and regressions all the time. You arrive at work in the morning, planning to write a new feature; but an upgrade has just arrived, and it needs a few changes to your project. You develop, test, and deploy these changes; in the meantime, another upgrade has just arrived, needing more changes, to something you had changed just a few days ago. The day ends, and you didn't even start to develop the new feature. You use more time chasing the upgrade stream than doing productive work. Long term releases "batch" the changes. When several changes affect one part of your software, you only have to deal with them once. Sometimes, you can even discard that part of your system and do something else, while with a continuous change stream, you might have wasted time adjusting little by little. A somewhat relevant post from Joel on Software: http://www.joelonsoftware.com/articles/fog0000000339.html http://www.joelonsoftware.com/articles/fog0000000339.html

207 points by luu 12y ago

PostgreSQL has a 5-year support window. I'm curious what negative consequences the author thinks PostgreSQL suffers as a result from that policy.

silvestrov 12y ago
It's a lot easier with a 5 year window for PostgreSQL: Kernel developers have to support all kinds of weird semi-buggy hardware that they can't buy in their country, they have to make the kernel work with badly written GPU drivers , and they have to deal with interrupt-level code which is notoriously difficult to debug.
- jeffdavis 12y ago
  The author's primary example for the "LTS is bad claim" was glibc, which is not a kernel. Are you saying the "LTS is bad claim" only applies to kernels and other things very close to the hardware (closer than a database)?

fidotron 12y ago

Upvoted for some interesting points, but it's quite wrong for the simple reason that on a commercial timescale one year is nothing. There are reasons Microsoft end up being dragged into supporting old OS versions, and it's because running an operating system is just a means to an end, which hopefully is doing something useful. "Upgrades" have, understandably, become synonymous with unnecessary pain and breaking things. Maybe OpenBSD really is more rigorous about quality control, but (as an example) if you were to just accept every Ubuntu update it wants to install you'd be wasting a significant amount of time just ensuring your system works properly.

deleted 12y ago
[deleted]
- specialp 12y ago
  This is implying that people actually want to upgrade full releases. Perhaps I would like to make a Linux box and have it last a few years, and only patch it for security fixes as it is working fine. Yes if I do have to upgrade that system 5 years later it is going to be more painful than every 6 months, but if I did it every 6 months I would have to devote development time and switch context every 6 months. Small updates are better if you have to upgrade entire systems, but many people do not want to risk destabilizing working systems every 6 months. For that we have LTS where someone will integrate in security fixes in a hopefully non impacting fashion and for that I am thankful
deathanatos 12y ago
> on a commercial timescale one year is nothing. I've worked in commercial environments where common invocations of `tar` did not work, because `tar` was a decade out of date. I had to learn things and habits that had died before I even started programming. It wasn't pleasant. Do not underestimate the age and stubbornness to upgrade of some environments. Recently, I've helped migrate software from Ubuntu Precise to Trusty, and the amount of differences make things mildly frustrating. We don't get to just run one or the other, and we can't just drop everything and move to Trusty. We have to continue to support the old while we build support for the new, briefly support both, transition to the new, then tear out the support for the old. Migrated. It's a lot of work when the changes are huge, but much more manageable when I can take the changes more piecemeal (it's one if statement, as opposed to many, that I need to manage at any given time). That's in a production environment. I run Gentoo at home. I much prefer its rolling releases to Ubuntu and Debian which I ran alongside and before, respectively. Things break every now and then. It's a tad annoying, but it gets fixed sooner or later.

zokier 12y ago

Isn't 6 month release cycle and support for past 2 releases essentially also the Fedora release model? From what I've heard (and used) that hasn't been massively successful. > No one upgrade is likely to end in disaster because there simply isn’t enough change for that to happen. That might be true for OpenBSD, but in Linux land the rate of change certainly is great enough to cause major breakage even at 6 month cycle.

craigds 12y ago

"Considered Harmful" Essays Considered Harmful: http://meyerweb.com/eric/comment/chech.html http://meyerweb.com/eric/comment/chech.html I think the OP is being a bit naïve if they expect all users to upgrade to a new OS every year. Upgrades of Ubuntu and OSX are usually quite painful endeavours, fraught with UI-breakage and new bugs to solve, and there's usually little incentive for users to upgrade in every 6-monthly release. IMHO Ubuntu has a good two-pronged approach - short-term support for most releases, but a LTS every couple of years for those who don't want to handle the pain of upgrading all the time. Anecdotally, even upgrading to stay up-to-date with the LTS versions can be difficult. A company I deal with is currently scrambling to ditch Ubuntu 10.04 before it loses support in April this year. That's 5 years old now. Companies don't upgrade for fun.

chollida1 12y ago

> A one year support window isn’t too short; it’s too long. Wow, I'm in very strong disagreement with the author on this point. I don't think I could find a single person, technical or otherwise who didn't use the same operating system for more than a year. In fact its the default for almost everyone who isn't technically inclined to upgrade their os only when they get a new computer. What about every single car produced today. I doubt you'd want to upgrade them every 6 months. As a ubuntu user, I find that I cross my fingers every time I run sudo apt-get upgrade. About half the time I get broken builds and once a year my OS will just flat out crash from it, or fail to reboot in virtual box. This view, while idealistic is just so laughable I can't begin to believe the author is serious. I've worked with some mission critical systems( stock exchange's) They take more than a year planning a big OS upgrade. To turn around and tell them that they should be doing this every 6 months is just so out of touch with reality, I don't know how to respond to the author:( Many companies won't install a new piece of software until its been proved out in production for 6 months to a year by someone else....

deleted 12y ago
[deleted]
- zokier 12y ago
  > That's what you need to do when OS releases are made only once every few years and are big. They don't have to be that way Ubuntu and Fedora releases happen already in 6 month cycles. This is not some novel unexplored concept.
  - wtallis 12y ago
    Ubuntu and Fedora are aggressive about incorporating new stuff. It's possible to have the same release cadence while operating with a more Debian-like attitude toward stability.
- taeric 12y ago
  I question the scale of just what is getting upgraded if it is "quick and painless." Note, that I do not mean to diminish the accomplishment. Especially as a fan of TeX and someone that often toutes that maybe we are learning the wrong software lessons by constantly rewriting everything to be in the latest safe language/paradigm. However, to not acknowledge all of the progress that is made in other areas is just as dangerous as not acknowledging the benefits of more deliberate methodologies.

zdw 12y ago

The downside of this is that OpenBSD is pretty difficult to upgrade in a clean manner given it's "uncompress the tarball over /, then clean stuff up manually" method. I really like OpenBSD, but it desperately needs a modern package management system. This is frankly my largest sticking point with it - it's faster and much cleaner in most cases to wipe, reinstall, then apply configuration than to try to upgrade.

deleted 12y ago
[deleted]
jkot 12y ago
How about debian with freebsd kernel?
- elektronjunge 12y ago
  The new pkg tool in freebsd is pretty good and meshes nicely with the ports tree. The marginal improvements for 3rd-party built packages with apt aren't really worth getting rid of the rest of freebsd for. It also doesn't help OpenBSD, which really needs a new package manager.
danielweber 12y ago
As someone who used to help maintain hundreds of OpenBSD boxes, yes. But don't forget, OpenBSD is designed for the developers.
brynet 12y ago
This is oft-repeated, but how is OpenBSD's package management system "not modern"? It handles upgrades properly, modified configuration in /etc is preserved. Packages are cryptographically signed as of 5.5, files are individually checksummed.. I'd say OpenBSD still handles dependencies and upgrades better than FreeBSD's supposedly "next-generation" pkg. OpenBSD's package system and ports tree are tightly integrated and work together, and despite the historic naming, OpenBSD's pkg_{add,delete,info,create}(1) are unrelated to the now deprecated FreeBSD equivalents.
12y ago

IshKebab 12y ago

Can we have a ban on "X considered harmful" titles? Have some originality.

jerf 12y ago

In theory, I agree. I've often likened code development to muscles; you get good at what you do, and what you don't do atrophies. If you push changes ever hour, you get very good at making sure that works. If you push them every three months, you get good at pushing them every three months, and panic when something requires you to move within days. And so on. This applies to all sorts of development process aspects (test a lot, get good at testing, fail to test, becomes impossible to test later, etc). In practice, neither what tedunangst nor I think matters, because once you ship something to a customer in any way, they're not going to upgrade it. Offer automated upgrades and they'll demand a way to turn them off. It's not just developer boxes we have to worry about, unfortunately.

gtirloni 12y ago

Oh please, all it would take for this "Ghost" vulnerability not to happen was someone at glibc to have made the right call regarding how to treat a buffer overflow in a function dealing with external data. They didn't, so what? The lesson here is that Red Hat & other companies should do additional review if they are shipping something branded as super secure and stable. Long term support is great. People screw up sometimes. Additionally, it's about time core software components get more attention from companies with the big bucks (and that are profiting from it).

vezzy-fnord 12y ago
Personally I'm of the idealistic belief that glibc is really starting to show its corrosion, and should be replaced by a clean-room, standards-compliant and elegant design like musl: http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc http://wiki.musl-libc.org/wiki/Functional_differences_from_g... Of course, this would be a massive change for the Linux ecosystem at large. It is my hope that projects like Sabotage Linux will eventually procure most major software to become musl-compatible, though. The source of the GHOST bug was in NSS, which musl lacks by design.
- GregBuchholz 12y ago
  Speaking of corrosion, maybe we should be building systems with a language which avoids most of the problems of C.
  - Silhouette 12y ago
    The trouble with that otherwise perfectly reasonable position is that no such language exists yet that doesn't have its own significant problems of one kind or another. I'm hopeful that this situation will finally change within the next decade and that C and C++ can be honourably discharged as soon as possible thereafter. However, I don't see this happening until newer languages with sounder foundations -- Rust comes to mind as a promising example -- have entered the mainstream. Until then, the ecosystem around C and C++ has too much momentum for a lot of projects to switch to more experimental technologies even if those technologies have advantages in terms of robustness.

zokier 12y ago

Personally I think a 6 month release cycle is the worst of both worlds. Either give me LTS and I'll do that one big upgrade every few years, or give me rolling release where any breakage should be relatively localized and minor. But with 6 month release cycle you have to constantly be upgrading, but the upgrades are still big "break the world" upgrades.

allendoerfer 12y ago

His reasoning makes sense but, as all developers tend to do, he spots an error in a system and wants to switch to another or invent a new one. You know the xkcd. Often times the solution is just to work together and fix the old one. For me the logical fix would be to patch all kinds of malicious, undefined or non-spec behaviors in LTS releases in short cycles regardless whether the developer thinks, it is security-critical or not. To make this more feasible you could either pay for it or use a minimal base system separated from the user-space. Both exists today.

JoachimSchipper 12y ago
That's both a lot of work and a lot of churn, though - people choose LTS because they don't want things to change!
- allendoerfer 12y ago
  In a sense, things do not change, they stop to be changed from a assumed state.

Sanddancer 12y ago

There are a lot of problems with this argument, especially when you go into the ports section. For example, the version of go between 5.4, 5.5, and 5.6 is different for all three versions, which means you're going to have to rebuild packages, run regression tests, etc every six months to make sure some new feature doesn't run into your code, or some bug isn't introduced. Similarly, in the case of the base install, you are still going to have to run a regression test there to ensure that any deprecated or removed features don't impact your workflow. These things are expensive from a development and ops perspective, and why most commercial software vendors tend to stick to a releasing for a few platforms they know are going to be supported for longer than a year or two. Yes, OpenBSD upgrades are mostly painless, but still, bugs can and do happen, and expecting everyone to be able to just drop everything and rebuild is plain and simple unreasonable.

learnstats2 12y ago

>Now on the one hand, this forces users to upgrade at least once per year. The problem with this article is this assumption which is not true in the slightest. There's no force which makes users upgrade - some users will upgrade when they're told; some will wait 5 or more years - and may have good reason to. > Nothing kills a bug report faster than “My network card worked in 4.4 but stopped working in 5.6.” Developers aren’t going to bisect five years of changes; you get to do that yourself. This statement can translate into don't ever upgrade. Is that really the goal?

sarciszewski 12y ago
No. Ideally, the goal is that users will upgrade as soon as a fix is available.
- taeric 12y ago
  I would think ideally, programmers wouldn't break things that are working. We know that ideal doesn't pan out.
  - sarciszewski 12y ago
    "programmers wouldn't break things that are working" So you want the world's best security experts to manually review every module of every program you use, as well as every piece of every operating system you want it to run on, to make sure there are 100% certainly no exploitable bugs that could require a fix down the line? That's the same argument, but with reality applied. Programmers don't "break things that are working", people find design flaws that could lead to remote exploitation. If you don't patch your stuff, enjoy being 0wned, not my problem. For the rest of the world, bleeding edge is the safest place to be.
    - taeric 12y ago
      Yes, yes I would love that. Though, as I noted, nobody expects that ideal to pan out. Reality applied to many large organizations to the prospect that things may break on an upgrade translates to "don't upgrade." And really, this isn't necessarily bad. So long as what isn't getting upgraded has been compartmentalized and does its job correctly. Consider, we don't even think about asking companies to upgrade processors on a yearly basis. Monitors? Good for years. Actually, pretty much all of the hardware. Why do we think software should be different?

jacquesm 12y ago

There is a cost to upgrading and there is a cost to staying behind. As soon as the cost to stay behind outweighs the cost to upgrade I upgrade, but no sooner. Otherwise I'd just be throwing away money and time. The bigger problem here is that the cost to upgrade is only known after the fact. What should be a routine job can easily spiral out of control into a marathon of misery.

thaumaturgy 12y ago

> Frequent upgrades amortize the cost and ensure that regressions are caught early. No one upgrade is likely to end in disaster because there simply isn’t enough change for that to happen. Oh, how I wish this were true. For what it's worth, it's pretty true as far as OpenBSD is concerned, in my experience. But OpenBSD is the exception here, not the rule. Everywhere else, developers all seem to have embraced "break early, break often". Eventually you get burned. For me, it was a routine should-have-been-minor web server update where one of the packages I relied on suddenly became unsupported and every single hosted site stopped working. Since there's no way to roll back server upgrades, I had a marathon night involving building a new server stack and migrating all hosted sites there by 8 a.m. But you can't yell at anybody when that happens, because the answer's always the same: it's not the developers' fault. Who really believes sysadmins wouldn't update everything all the time if they could? Old, dodgy, out-of-date servers exist exactly because updates are butthole-puckering, because everyone's been burned at least once by a "minor" update, and because once the damage is done, undoing it is horrifyingly difficult.

edofic 12y ago
"Since there's no way to roll back server upgrades" It is if you run a modern filesystem like ZFS or btrfs. You just do a cheap snapshot before upgrading(can be automated) and roll back if there are problems. Even works with lvm.
- thaumaturgy 12y ago
  Ooh, thanks. That's a good idea. I'd never considered running ZFS on my small servers.
  - keithpeter 12y ago
    Or perhaps simply gzip of the / partition before upgrading. If insuperable problems, zap new / and restore old known working one? Assuming data/sites are on different partitions.
    - 12y ago

Silhouette 12y ago

This is not an argument against long term support. It is an argument against using OpenBSD for any project you need to work for more than one year.

CrLf 12y ago

Forcing users to upgrade frequently has one outcome, and one outcome only: they will stop upgrading, period. If you give users only two choices, they will choose one, but the one that requires less effort. There is more to life than keeping up with the constant upgrade treadmill. It's already bad as it is. Some Linux distributions are on a 3-year support cycle, which is too short. If you have many servers, especially many servers running a lot of different things, upgrading every 3-years means having people doing nothing more than upgrades. Once you fix your applications to support the changes in the next OS release, that release is almost out of support...

wmf 12y ago
they will stop upgrading, period Has that been observed with Chrome, ChromeOS, iOS, etc.?
- CrLf 12y ago
  That's an apples to oranges comparison. To upgrade Chrome a consumer depends only on himself. To upgrade a server OS, the "user" (the sysadmin) depends on a number of other people with different priorities. And that's _if_ the people he depends on are still with the company... I've been faced inumerous times with impossible upgrades (nobody cares about upgrading besides you, so nobody cares to fix whatever is preventing the upgrade). And when the upgrade is possible, many times it takes years. That same user that upgrades Chrome on his laptop, is still stuck with IE7 at work because his company relies on it to pay his salary.

slasaus 12y ago

It's funny that just today, before I found out about the glibc vulnerability, I reconsidered if I really want to upgrade my Ubuntu 10.04 mail and web servers to Ubuntu 14.04 LTS. I was triggered after reading some bad things about 14.04 [1]. I've looked at the M:Tier binpatches and package upgrades for OpenBSD, looked at FreeBSD, Debian with it's experimental LTS, but eventually I'm still in favor of the 5-year support for Ubuntu server. I have bad experiences in upgrading production machines in-place, whether it is OpenBSD, Ubuntu or Debian and always install a new machine (vps) side-by-side which is really the only stress-free guaranteed way to go in my opinion. Having to do this only once every 5 years is really a lot nicer then at least once a year. The good security backports of Ubuntu, minimal breakage (auto-security upgrades at my Ubuntu servers have been working almost flawlessly [2]) are the least maintenance, stable and secure setup I can imagine. OpenBSD having only one year of support, no binpatches of the kernel and having no stable security fixes of the packages are the reason I only use it with anything that can be done by the base system (backup host and nameserver). OpenSMTPd looks very promising, but I would need supported amavisd packages, same goes for httpd that needs php in my setup. Besides it's limited use, I still love OpenBSD and the mindset that stewards it. If only they had longer support and binpatches for kernel and packages :) [1] https://tim.siosm.fr/blog/2014/04/25/why-not-ubuntu-14.04-lts/ https://tim.siosm.fr/blog/2014/04/25/why-not-ubuntu-14.04-lt... [2] the mail config was overwritten once after an auto security update of dovecot in 10.04, quickly recovered it with etckeeper (/etc in git)

tghw 12y ago
Thanks for the tip on etckeeper! I've been wanting something like that for a while.

coherentpony 12y ago

"[Useful popular thing] considered harmful."

jrochkind1 12y ago

The only thing that would make this realistic is if open source developers prioritize backwards compatibility very highly. Of course, there will always be some bugs anyway. And it varies across language (or other open source affinity) communities. But in many communities, developers aren't really even _trying_ to ensure backwards compatibility. If developers committed to backwards compatibiilty with successive versions for X number of years -- then you could update to the latest release for a number of years worrying less about breaking things, rather than needing to freeze the versions you use for that number of years. Yes, this would significantly increase developer hours on open source projects. Nothing comes for free. But wouldn't those developer hours be better spent centrally on ensuring backwards compatibility in the projects themselves; as opposed to every developer of every consumer of the project needing to deal with backwards incompatibility on every upgrade? Or every distro needing to backport patches? (This latter is more debatable, which is exactly why we have the status quo)

kbenson 12y ago

Every time this topic comes up, we have people talking about really different things not clearly explaining what they are referring to, and causing confusion and people not understanding the reasoning of others. I've read the following arguments already, but with portions of them implicit. See if you can determine why the people are talking at cross points: - "I generate VM server images for all my needs, and deploy to virtualization infrastructure behind load balancers to handle services. I treat he OS like an application like the rest of the stack, and all my data is abstracted to a data layer. I just generate a new image with patches and test it, then deploy if it works in my test suite." - "I have hundreds of servers in dozens of roles with different software needs, and I need them to be secure and stable in a timely fashion, and I can't spend multiple weeks achieving that. Long term support and back-patching allows me the time to plan needed large changes in infrastructure without having to spend all my time managing updates, software changes and configuration changes multiple steps down the dependency graph." - "I have a few servers with a few roles, or many servers with one or two roles, and I can manage frequent updates just fine, and it allows me to take advantage of the newest features, get security updates immediately as the software authors fix it, and I don't have to worry about end of life of software." - "I have an application stack with multiple dependencies, and I just make sure to update my stack as I make changes to the software. I would love/have set up continuous integration software to build and test everything as I go, so I know if it works or not before taking it live." As someone who's been in all of these situations at different points, often multiple at once, let me be clear: Unless you have an argument that addresses ALL of these situations, you really haven't thought through the issue.

hawleyal 12y ago
I think the author is advocating that those scenarios should converge. His argument is that second scenario is an anti-pattern and should be corrected to be more inline. It might be a pipe dream, but it's still his perspective.
- kbenson 12y ago
  They serve different needs. The second scenario may well be better served by moving towards the first, but that's not a quick project, especially when uptime and reliabiity is important. Even with that, the for the third scenario a full VM infrastructure may by overkill. This whole topic can be summed up with "Examine your needs, examine your options, make the best choice you can at the time, try to make your life easier by leaving pathways to change your choice with the least problems based on expected future needs." Just because someone looks to be doing something similar to you, doesn't mean their needs and constraints aren't quite different. I think this is the big thing most people miss.

contingencies 12y ago

Reading the title I assumed this was a rant about how proper ops processes replaced the need for LTS releases and the demonstrably bad practices they could be seen to encourage. IMHO, this would have been a more interesting and valid point. The current post is a straw man argument for a few reasons. First, because vulnerabilities will be exploited whether the window of availability is 1 day, 100 days, or 1000 days (like this one) and likewise, upgrades will be missed, for various reasons, whether or not they are encouraged. Second, the author spuriously implies that a 6 month sliding upgrade window ("one size fits all") approach is superior, easier and more appropriate for everyone through a combination of broad assumption and vague hand-waving about developer time efficiency. Rubbish. If you want newer code, then ASAP is the go and BSD-style monolithic releases are bad... something closer to the versionless OS ideal of constant, incremental, package-wise release processes such as Gentoo's portage (BSD-inspired!) where you can even install a -9999 (git HEAD) version of many packages would be ideal. Obviously, with such an approach, stability caveats must be considered (as they are). Thus, both FreeBSD and OpenBSD actually represent a 'middle-ground' between Gentoo and commercial Linux vendor LTS, with OS wide release processes incorporating some greater collective package and package interoperability testing, combined with the package-wise release of things through the ports trees. What's the real ideal here? A general purpose means to build working, tested, maintainable, secure systems with minimal effort for a broad audience. How to get there? Clearly, not by whinging about external release window frequencies of volunteer-based projects, as any frequency still creates bugs, and no process is perfect. The answer, I believe, is time-honored. Careful design for failure and minimalism, and good process. Test. Measure. Remove surplus features and components. Iterate.

ak217 12y ago

Spoken like someone who has never had to maintain a large production system. This notion is handily disproved by the market, though. Red Hat and Canonical enjoy commercial success and mindshare precisely because they provide stable long-term support platforms on which others can build software. Most of Canonical's worth is actually embodied in their commitment to LTS releases, which provide a sweet spot between stability and the glacial pace of RHEL.

reality_czech 12y ago

Yes, we should all step forward into the wonderful world of running brand new software all the time in production. I'm sure it will all be much more secure than relying on year old software that is being actively maintained. Is the date on this one supposed to be April 1st?

cesarb 12y ago

A problem with the lack of long term support is that you have to keep moving. The most important part of "long term support" distributions is that breaking changes are kept to a minimum. If a "long term support" distribution releases an update to for instance glibc, you can expect that applying that update will change nothing other parts of your software stack might depend on. The dependencies might be subtle; for instance, a new version of a database server might have optimized its query planner, which happened to make one particular query your software does a couple of percent slower, which led to it using a few more seconds to do its processing, enough to push it over the timeout limit for a different part of the system. So the only sane way to avoid breaking changes is to avoid all changes. The opposite would be, as advocated in this post, frequent upgrades. "Frequent upgrades amortize the cost and ensure that regressions are caught early", but that means you are dealing with upgrades and regressions all the time. You arrive at work in the morning, planning to write a new feature; but an upgrade has just arrived, and it needs a few changes to your project. You develop, test, and deploy these changes; in the meantime, another upgrade has just arrived, needing more changes, to something you had changed just a few days ago. The day ends, and you didn't even start to develop the new feature. You use more time chasing the upgrade stream than doing productive work. Long term releases "batch" the changes. When several changes affect one part of your software, you only have to deal with them once. Sometimes, you can even discard that part of your system and do something else, while with a continuous change stream, you might have wasted time adjusting little by little. A somewhat relevant post from Joel on Software: http://www.joelonsoftware.com/articles/fog0000000339.html http://www.joelonsoftware.com/articles/fog0000000339.html