5 ms·

I generally tow the line when it comes to software, but I do have a couple contrarian opinions and one of them is that LTS is really, really dumb. Not just for

I generally tow the line when it comes to software, but I do have a couple contrarian opinions and one of them is that LTS is really, really dumb. Not just for maintainers, for everyone. LTS destroys your codebase. LTS allows mistakes like pushing state into your server that you didn't know was there. LTS makes it much harder to upgrade because by the time you're moving off of, say, Ubuntu 12.04 the whole world has changed. LTS multiplies the number of things that can go wrong. LTS makes people think that the OS is going to be around forever so they don't architect their devices to be OS-updateable. IoT is bad now, but it's slowly getting worse too because half these devices are stuck on some out of date version of linux running some out of date version of Python. All my servers run Ubuntu 17.10. Rails is on the latest version. Ruby is on the latest version. I don't pin any of my gems and every time I upgrade Ruby I upgrade the latest version of all the gems too. This is only possible with extensive testing, and mistakes do get made here and there, but overall it's much, much easier handling these issues one at a time. The only thing I don't do is keep current on packages that I install via `apt` because I don't want to risk building from source.

Kenji 9y ago
If you need the latest OS version for your software to run, you're doing software design wrong. Except for very special features, software should run all the way to 10-20 years old OS versions. And if it doesn't, you should have abstractions in place such that all you have to do is change a dll and be good to go again. Why the hell would a high level language like Ruby have such deep OS dependencies? Use the goddamn POSIX standard and you'll be compatible to every single important OS that's come out in the last decades, except for Windows which can be made POSIX compatible with a few layers of abstraction.
pjc50 9y ago
> IoT is bad now, but it's slowly getting worse too because half these devices are stuck on some out of date version of linux running some out of date version of Python. Sure, this is bad - but if they are to be OS updateable then the OS has to guarantee it won't drop support for things, or increase significantly in size because then it won't for in the flash. Enterprise customers really like LTS as a concept, but they should be made to pay for it like the article says.
- 3pt14159 9y ago
  I just think you have a choice of one or neither of these, but not both: 1. Internet connected. 2. Code and forget. Choose one or none, but not both. Selling medical equipment that gets hacked by wannacry is not acceptable. It isn't even theoretically possible to promise that the OS won't drop support for things since you may rely on something that is later exploitable. Realistically speaking, unless you're writing code for something crazy like VR, there is a great chance that with tests you can update the code to match the new OS in a week or two.
blueflow 9y ago
Using Ubuntu has the big advantage that you dont have to maintain the packages yourself, so you just dont notice all the regressions that maintainer have to deal with on a daily basis. Seeing fundamental things like glibc or gcc breaking almost every release makes you start to value LTS releases a bit more. Disclaimer: Im maintainer myself and i feel your opinion is disrespectful towards the people who fix regressions for you (for free, even!).
- 3pt14159 9y ago
  There is no disrespect. I don't disrespect hardworking people trying to make the world better. This is just what I've found after writing software for decades. It's a viewpoint that I'm willing to change, but also one I think others should entertain so that all our software gets better and stays current. Also, as I said in the original comment, I only lean on apt for a minority of things. Most stuff gets upgraded right away and you know what? I find and fix or communicate bugs because I'm on the bleeding edge. And I frankly do not see how my using apt invalidates my views on LTS. In the ruby world upgrading is almost never a pain because most of us keep current and test our codebases. I shouldn't be maintaining packages myself, and I do think that if the people behind Debian / Ubuntu / rando packages available took my worldview it would be easier to always be on the latest version and upgrading shit wouldn't be this giant headache all the time. Common, glibc should break or break something every release. This is insanity.
  - derefr 9y ago
    In language package ecosystems (like Ruby's), you decide when to push out a new release. You find that updating your lockfile breaks your code, so you put it off. You push out a few versions without updating the deps. Then, in one release, you update them all, fix all the problems that introduces, and then push out a new version that has the new deps and works with them. OS package management doesn't work this way, because, for the most part, for almost the entire history of OSes, developers have had no control over the environment their application is running under. In a lot of cases (enterprise IT; academic/HPC mainframes; PaaSes) someone else has an upgrade schedule for the OS packages, and you don't get any say on when it happens. The OS updates, your package breaks, your problem. You can maybe ask the ops department to hold off on updating, or have your PaaS "pin" your stack (until it becomes deprecated.) Either way, you're now vulnerable to security vulnerabilities while (hurriedly) working to push out the new release so that ops can get back to doing its job as normal. LTS is a guarantee by the OS distributor that your ops staff can update any package at any time (from the LTS repos, at least), and such updates won't break your deployed custom software. So LTS solves the cases above: enterprise/academic/HPC/PaaS ops staff can just upgrade as they will, without worry that anyone will tell them to hold off. But the value of LTS goes even further: if updates are guaranteed by the distributor to not break your runtime environment, then you can actually allow OS package upgrades to happen completely automatically and asynchronously to your development release schedule. The OS can just update itself. The OS distributor is now, themselves, part of your ops staff, ensuring that these updates aren't bothering anyone downstream of them. Your actual company doesn't have to care about the OS. It's always going to be secure, because it's going to update as soon as vulnerabilities are discovered+patched, and you can just deploy your code at your own pace without worry. Now, mind you, this all might sound quaint to people who bake containers for everything and treat said containers' host OS as immutable infrastructure. For these people, OS package security vulnerabilities aren't really something you can let "fix itself"; they're something that requires baking a new version of your container-image, just as app library-dependency security vulnerabilities would be. Relying on OS services rather than bundling a copy into your container-image is a trade-off in this case: you don't get to pin the version of a dependency to ensure its exact behaviour, but in exchange, the distro will attempt to ensure that it will be free of security vulnerabilities while being pretty much the same version you started with. (Oh, and as a side-benefit, usually LTS package updates don't force system reboots. Recently even kernel updates don't, with fancy ksplice-based patch distribution channels. So LTS releases also mean long individual-host uptimes, which is handy if you're running a service on the cheap and can't afford much horizontal scaling to manage load-balancing under rolling updates, but want five or more 9s of availability anyway. Try getting that using immutable-infrastructure reboot-to-update VM images.)
bluedino 9y ago
>> I don't pin any of my gems and every time I upgrade Ruby I upgrade the latest version of all the gems too. How do you deal with this in production? Gems get abandoned. Some gems get upgraded faster than others. Some before you're ready. You get into a race where you either need to fix your code, or fix a gem, or not upgrade. What do you do then? What if you don't have time to fix code to deal with a gem upgrade that ca happen at any time?
- 3pt14159 9y ago
  What do you do when a zero day comes out for nginx or openssl? When things somehow go sideways I do the same thing: Triage the situation, including taking things offline if needed, make choices that solve the short term situation, run the tests, everything ok push live. Otherwise mitigate the issue by partially removing services (no image upload right now) and getting the rest of them live. On the server itself I have a second copy of the repo. I install the new version of ruby in that repo and install the gems in that repo. I run the tests there and I check to make sure that Gemfile.lock hasn't changed. All good? Time to go live. Otherwise? Fix things locally first and re-push.
bpicolo 9y ago
LTS would be a ton less necessary if people took backwards compat as seriously as some do. Rich Hickey has a great talk here https://www.youtube.com/watch?v=oyLBGkS5ICk https://www.youtube.com/watch?v=oyLBGkS5ICk That said, doesn't work at all levels, but in general it's not nearly common enough in the ecosystem to keep compat consistent. When I think of libraries I love to use, I think mostly of libraries where compat is very well maintained. Package management systems tend to be working against you here. Bumping major versions for "good" reasons makes it so that your biggest library users just get stuck, because they have so much code in the old paradigm. Real compat changes deserve new names/namespaces, so that you can progressively upgrade.
- moomin 9y ago
  Backwards compatibility is exactly why clojure.set’s JIRA is full of won’t fixes, though. It’s not just a cost for maintainers.
  - bpicolo 9y ago
    Yeah, for sure. I think the best takeaway for me from that talk was that major versioning is actually a kinda crummy part of dependency managers, because the way we structure them / code tends to allow you to only use one version of some library at a time. I've definitely been bit on many occasions by libraries doing entire restructures of the whole thing and then pushing under the same namespace. A better strategy for namespacing would go a long way. The clojure.set thing is crummy but a hard call. That's a super fundamental piece of the library puzzle, and write once, run forever is a really very expected thing there. Those are something that a wrapper can overcome pretty simply though. In a lot of those cases the approach they take does seem reasonable though, namely that e.g. clojure.set/union perform exactly as is documented on the documented inputs.
tomc1985 9y ago
In my systems integration work I would much rather have LTS than not. LTS is a blessing because I would rather front-load upgrade issues all at once than have them spread out over a few years because we shipped products on a bleeding-edge OS. LTS helps administrators and integrators maintain reliability, which is very important in the perception of one's (at least our) users. More important than keeping on the bleeding edge.

Kenji 9y ago

If you need the latest OS version for your software to run, you're doing software design wrong. Except for very special features, software should run all the way to 10-20 years old OS versions. And if it doesn't, you should have abstractions in place such that all you have to do is change a dll and be good to go again. Why the hell would a high level language like Ruby have such deep OS dependencies? Use the goddamn POSIX standard and you'll be compatible to every single important OS that's come out in the last decades, except for Windows which can be made POSIX compatible with a few layers of abstraction.

pjc50 9y ago

> IoT is bad now, but it's slowly getting worse too because half these devices are stuck on some out of date version of linux running some out of date version of Python. Sure, this is bad - but if they are to be OS updateable then the OS has to guarantee it won't drop support for things, or increase significantly in size because then it won't for in the flash. Enterprise customers really like LTS as a concept, but they should be made to pay for it like the article says.

3pt14159 9y ago
I just think you have a choice of one or neither of these, but not both: 1. Internet connected. 2. Code and forget. Choose one or none, but not both. Selling medical equipment that gets hacked by wannacry is not acceptable. It isn't even theoretically possible to promise that the OS won't drop support for things since you may rely on something that is later exploitable. Realistically speaking, unless you're writing code for something crazy like VR, there is a great chance that with tests you can update the code to match the new OS in a week or two.

blueflow 9y ago

Using Ubuntu has the big advantage that you dont have to maintain the packages yourself, so you just dont notice all the regressions that maintainer have to deal with on a daily basis. Seeing fundamental things like glibc or gcc breaking almost every release makes you start to value LTS releases a bit more. Disclaimer: Im maintainer myself and i feel your opinion is disrespectful towards the people who fix regressions for you (for free, even!).

3pt14159 9y ago
There is no disrespect. I don't disrespect hardworking people trying to make the world better. This is just what I've found after writing software for decades. It's a viewpoint that I'm willing to change, but also one I think others should entertain so that all our software gets better and stays current. Also, as I said in the original comment, I only lean on apt for a minority of things. Most stuff gets upgraded right away and you know what? I find and fix or communicate bugs because I'm on the bleeding edge. And I frankly do not see how my using apt invalidates my views on LTS. In the ruby world upgrading is almost never a pain because most of us keep current and test our codebases. I shouldn't be maintaining packages myself, and I do think that if the people behind Debian / Ubuntu / rando packages available took my worldview it would be easier to always be on the latest version and upgrading shit wouldn't be this giant headache all the time. Common, glibc should break or break something every release. This is insanity.
- derefr 9y ago
  In language package ecosystems (like Ruby's), you decide when to push out a new release. You find that updating your lockfile breaks your code, so you put it off. You push out a few versions without updating the deps. Then, in one release, you update them all, fix all the problems that introduces, and then push out a new version that has the new deps and works with them. OS package management doesn't work this way, because, for the most part, for almost the entire history of OSes, developers have had no control over the environment their application is running under. In a lot of cases (enterprise IT; academic/HPC mainframes; PaaSes) someone else has an upgrade schedule for the OS packages, and you don't get any say on when it happens. The OS updates, your package breaks, your problem. You can maybe ask the ops department to hold off on updating, or have your PaaS "pin" your stack (until it becomes deprecated.) Either way, you're now vulnerable to security vulnerabilities while (hurriedly) working to push out the new release so that ops can get back to doing its job as normal. LTS is a guarantee by the OS distributor that your ops staff can update any package at any time (from the LTS repos, at least), and such updates won't break your deployed custom software. So LTS solves the cases above: enterprise/academic/HPC/PaaS ops staff can just upgrade as they will, without worry that anyone will tell them to hold off. But the value of LTS goes even further: if updates are guaranteed by the distributor to not break your runtime environment, then you can actually allow OS package upgrades to happen completely automatically and asynchronously to your development release schedule. The OS can just update itself. The OS distributor is now, themselves, part of your ops staff, ensuring that these updates aren't bothering anyone downstream of them. Your actual company doesn't have to care about the OS. It's always going to be secure, because it's going to update as soon as vulnerabilities are discovered+patched, and you can just deploy your code at your own pace without worry. Now, mind you, this all might sound quaint to people who bake containers for everything and treat said containers' host OS as immutable infrastructure. For these people, OS package security vulnerabilities aren't really something you can let "fix itself"; they're something that requires baking a new version of your container-image, just as app library-dependency security vulnerabilities would be. Relying on OS services rather than bundling a copy into your container-image is a trade-off in this case: you don't get to pin the version of a dependency to ensure its exact behaviour, but in exchange, the distro will attempt to ensure that it will be free of security vulnerabilities while being pretty much the same version you started with. (Oh, and as a side-benefit, usually LTS package updates don't force system reboots. Recently even kernel updates don't, with fancy ksplice-based patch distribution channels. So LTS releases also mean long individual-host uptimes, which is handy if you're running a service on the cheap and can't afford much horizontal scaling to manage load-balancing under rolling updates, but want five or more 9s of availability anyway. Try getting that using immutable-infrastructure reboot-to-update VM images.)

bluedino 9y ago

>> I don't pin any of my gems and every time I upgrade Ruby I upgrade the latest version of all the gems too. How do you deal with this in production? Gems get abandoned. Some gems get upgraded faster than others. Some before you're ready. You get into a race where you either need to fix your code, or fix a gem, or not upgrade. What do you do then? What if you don't have time to fix code to deal with a gem upgrade that ca happen at any time?

3pt14159 9y ago
What do you do when a zero day comes out for nginx or openssl? When things somehow go sideways I do the same thing: Triage the situation, including taking things offline if needed, make choices that solve the short term situation, run the tests, everything ok push live. Otherwise mitigate the issue by partially removing services (no image upload right now) and getting the rest of them live. On the server itself I have a second copy of the repo. I install the new version of ruby in that repo and install the gems in that repo. I run the tests there and I check to make sure that Gemfile.lock hasn't changed. All good? Time to go live. Otherwise? Fix things locally first and re-push.

bpicolo 9y ago

LTS would be a ton less necessary if people took backwards compat as seriously as some do. Rich Hickey has a great talk here https://www.youtube.com/watch?v=oyLBGkS5ICk https://www.youtube.com/watch?v=oyLBGkS5ICk That said, doesn't work at all levels, but in general it's not nearly common enough in the ecosystem to keep compat consistent. When I think of libraries I love to use, I think mostly of libraries where compat is very well maintained. Package management systems tend to be working against you here. Bumping major versions for "good" reasons makes it so that your biggest library users just get stuck, because they have so much code in the old paradigm. Real compat changes deserve new names/namespaces, so that you can progressively upgrade.

moomin 9y ago
Backwards compatibility is exactly why clojure.set’s JIRA is full of won’t fixes, though. It’s not just a cost for maintainers.
- bpicolo 9y ago
  Yeah, for sure. I think the best takeaway for me from that talk was that major versioning is actually a kinda crummy part of dependency managers, because the way we structure them / code tends to allow you to only use one version of some library at a time. I've definitely been bit on many occasions by libraries doing entire restructures of the whole thing and then pushing under the same namespace. A better strategy for namespacing would go a long way. The clojure.set thing is crummy but a hard call. That's a super fundamental piece of the library puzzle, and write once, run forever is a really very expected thing there. Those are something that a wrapper can overcome pretty simply though. In a lot of those cases the approach they take does seem reasonable though, namely that e.g. clojure.set/union perform exactly as is documented on the documented inputs.

tomc1985 9y ago

In my systems integration work I would much rather have LTS than not. LTS is a blessing because I would rather front-load upgrade issues all at once than have them spread out over a few years because we shipped products on a bleeding-edge OS. LTS helps administrators and integrators maintain reliability, which is very important in the perception of one's (at least our) users. More important than keeping on the bleeding edge.