5 ms·

I think one of the good things to come out of this may be an increased sense of conservatism around upgrading. Far too many people, including developers, seem t

I think one of the good things to come out of this may be an increased sense of conservatism around upgrading. Far too many people, including developers, seem to just accept upgrades as always-good instead of carefully considering the risks and benefits. Raising the bar for accepting changes can also reduce the churn that makes so much software unstable.

sebstefan 3y ago
All things considered I'm not sure that'd be such a good thing How many security issues spring from outdated packages vs packages updated too hastily?
- denimnerd42 3y ago
  and when you do get a security issue and you're using a 10 year old version the upgrade is going to be really really difficult vs incremental upgrades when they are available. or are you going to fork and assume responsibility for that library code too?
- Hackbraten 3y ago
  On top of that: A newly-introduced security issue tends to have very limited exploitability, because it's valuable, not-yet well understood, and public exploits are yet to be developed. Compare to that a similar vulnerability in an older package: chances are that everything about it has been learned and is publicly known. Exploits have become a commodity and are now part of every offensive security distro on the planet. If you run that vulnerable version, there's a real risk that a non-targeted campaign will randomly bite you.
  - maerF0x0 3y ago
    > valuable, not-yet well understood, and public exploits Except in the scenario that is this exact case: Supply chain attacks that are developed with the exploit in mind.
    - Hackbraten 3y ago
      I agree in principle. But even if the backdoor is deliberate (as is the case here), there’s limited risk for the average person. Nobody in their right mind is going to attack Jane Doe and risk burning their multi-million dollar exploit chain. For an old vulnerability, however, any unpatched system is a target. So the individual risk for the average unpatched system is still orders of magnitude higher than in the former scenario.
cesarb 3y ago
> Far too many people, including developers, seem to just accept upgrades as always-good instead of carefully considering the risks and benefits. Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability, since it was introduced early in the 2.x series. The old 1.x log4j versions had other known vulnerabilities, but only if you were using less common appenders or an uncommon server mode or a built-in GUI log viewer (!); the most common use of log4j (logging into a local file) was not exposed to any of these, and in fact, you could remove the vulnerable classes and still have a functional log4j setup (see for instance https://www.petefreitag.com/blog/log4j-1x-mitigation/ https://www.petefreitag.com/blog/log4j-1x-mitigation/ which I just found on a quick web search). Did log4shell (and a later vulnerability which could only be exploited if you were using Java 9 or later, because it depended on a new method which was introduced on Java 9) lead people to question whether always being on the "latest and greatest" was a good thing? No, AFAIK the opposite happened: people started to push even harder to keep everything on the latest release, "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".
- JohnMakin 3y ago
  > Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability Lol, this exact thing happened at my last gig. When I first learned of the vulnerability I panicked, until I found out we were so outdated it didn't affect us. We had a sad laugh about it. > "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy". I think there is some truth to this motivation though - if you are on an ancient 1.X version and have to jump a major version of two, that almost always causes pain depending on how critical the service or library is. I don't pretend to know the right answer but I always tend to wait several versions before upgrading so any vulnerabilities or fixes can come by the time I get to the upgrade.
  - cesarb 3y ago
    A lot of people were in that exact same situation. So many, that the original author of log4j 1.x released a fork to allow these people to keep using the old code while technically being "up to date" and free of known vulnerabilities: https://reload4j.qos.ch/ https://reload4j.qos.ch/
TheKarateKid 3y ago
About a decade ago, the industry shifted from slow, carefully evaluated infrequent updates (except for security) to frequent, almost daily updates that are mandatory. I'd say this was pioneered by the Chromium team and proved to be beneficial. The rest of the industry followed. Now we're in a position where most projects update so quickly, that you don't really have a choice. If you need to update one component, there's a good chance it will require that many more as dependencies in your project will require an update to be compatible. The industry as a whole sacrificed stability and some aspects of security for faster advancements and features. Overall I'd say the net benefit is positive, but its times like these that remind us that perhaps we need to slow things down just a little and do a bit of a course correction to bring things into a better balance.
nilsherzig 3y ago
Wouldn't that just result in exploits written for old versions? A successful exploit for something that everyone is running might be worse, than a backdoor on blending edge systems. Everyone being on different versions results in something like a moving target
paulmd 3y ago
well, it's the bazaar vs the cathedral, isn't it? bazaar moves a lot faster. Everyone likes that part, except when it breaks things, and when they have to chase an upstream that's constantly churning, etc. but most people don't consider that a cathedral itself might have some engineering merit too. cathedrals are beautiful and polished and stable. I highly encourage people to try freeBSD sometime. Give ports a try (although the modern sense is that poudrie is better even if you want custom-built packages). See how nicely everything works. All the system options you need go into rc.conf (almost uniformly). Everything is documented and you can basically operate the system out of the FreeBSD Handbook documentation (it's not at all comparable to the "how to use a window or a menu" level intro stuff the linux provides). You can't do that when everything is furiously churning every release. everything just works, everything is just documented, it's an experience when you're coming from linux. and that forum post from 2007 on how to tweak a service script is probably still valid, because BSD hasn't had 3 different init systems over that timespan etc. just like "engineering is knowing how to build a bridge that barely doesn't fall over", engineering here is knowing what not to churn, and fitting your own work and functionality extensions into the existing patterns etc. like it doesn't have to be even "don't make a bigger change than you have to", you just have to present a stable userland and stable kernel interface and stable init/services interface. the fact that linux doesn't present a stable kernel interface is actually fairly sketchy/poor engineering, it doesn't have to be that way, a large subset of kernel interfaces probably should be stable.
Vicinity9635 3y ago
I'm kinda the opposite. Way too many times I've seen "upgrades" actively remove things I liked and add things I hate. I hold off on letting mobile apps update because they almost always get worse, not better.
- hinkley 3y ago
  I think it was Ted Neward who argued that libraries should come with a core and ancillary packages. Only the stuff really necessary and broadly popular goes into the base library, and all of the low use features should go into one or more ancillary libraries. I still think this is the way to go, but it does move some of the problem around in that you have to pick good cleave lines such that people aren’t forced to pull 5 libraries in for 5 features they need. You want one software domain to use two or three and another to use an overlapping set of three.
cryptonector 3y ago
That's a double-edged sword. What happens when you need to upgrade in order to get vulnerability fixes?
MySweetHubert 3y ago
The 2017 WannaCry ransomware attack would be a good counter example, the virus spread even though it was already fixed in an update from MS a bit more than a month before.
nickm12 3y ago
I don't believe this will result in conservatism around upgrading nor that it would be good if it did. The vast, vast majority of security vulnerabilities are due to regular bugs and once the details are known, they can be exploited. Not patching leaves you open to these exploits. Supply chain attacks are a real threat too, but being conservative about upgrading to improve your security posture is like saying "I'm worried someone with the key to my house might rob me, so I'm going to remove the lock from my door."
thegrim000 3y ago
This drove me crazy at previous companies where our build system dependencies were specified with wildcards so they would just auto pull in the latest version whenever you built the project. Not only are there security issues with that, as seen here, but it destroys your ability to have deterministic builds; every time you build the project it might be building against a different set of dependencies. You lose the ability to even compare builds against each other, because even though the builds are for the same exact project code, the dependency binaries might be different, and so the two builds could run differently.

sebstefan 3y ago

All things considered I'm not sure that'd be such a good thing How many security issues spring from outdated packages vs packages updated too hastily?

denimnerd42 3y ago
and when you do get a security issue and you're using a 10 year old version the upgrade is going to be really really difficult vs incremental upgrades when they are available. or are you going to fork and assume responsibility for that library code too?
Hackbraten 3y ago
On top of that: A newly-introduced security issue tends to have very limited exploitability, because it's valuable, not-yet well understood, and public exploits are yet to be developed. Compare to that a similar vulnerability in an older package: chances are that everything about it has been learned and is publicly known. Exploits have become a commodity and are now part of every offensive security distro on the planet. If you run that vulnerable version, there's a real risk that a non-targeted campaign will randomly bite you.
- maerF0x0 3y ago
  > valuable, not-yet well understood, and public exploits Except in the scenario that is this exact case: Supply chain attacks that are developed with the exploit in mind.
  - Hackbraten 3y ago
    I agree in principle. But even if the backdoor is deliberate (as is the case here), there’s limited risk for the average person. Nobody in their right mind is going to attack Jane Doe and risk burning their multi-million dollar exploit chain. For an old vulnerability, however, any unpatched system is a target. So the individual risk for the average unpatched system is still orders of magnitude higher than in the former scenario.

cesarb 3y ago

> Far too many people, including developers, seem to just accept upgrades as always-good instead of carefully considering the risks and benefits. Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability, since it was introduced early in the 2.x series. The old 1.x log4j versions had other known vulnerabilities, but only if you were using less common appenders or an uncommon server mode or a built-in GUI log viewer (!); the most common use of log4j (logging into a local file) was not exposed to any of these, and in fact, you could remove the vulnerable classes and still have a functional log4j setup (see for instance https://www.petefreitag.com/blog/log4j-1x-mitigation/ https://www.petefreitag.com/blog/log4j-1x-mitigation/ which I just found on a quick web search). Did log4shell (and a later vulnerability which could only be exploited if you were using Java 9 or later, because it depended on a new method which was introduced on Java 9) lead people to question whether always being on the "latest and greatest" was a good thing? No, AFAIK the opposite happened: people started to push even harder to keep everything on the latest release, "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".

JohnMakin 3y ago
> Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability Lol, this exact thing happened at my last gig. When I first learned of the vulnerability I panicked, until I found out we were so outdated it didn't affect us. We had a sad laugh about it. > "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy". I think there is some truth to this motivation though - if you are on an ancient 1.X version and have to jump a major version of two, that almost always causes pain depending on how critical the service or library is. I don't pretend to know the right answer but I always tend to wait several versions before upgrading so any vulnerabilities or fixes can come by the time I get to the upgrade.
- cesarb 3y ago
  A lot of people were in that exact same situation. So many, that the original author of log4j 1.x released a fork to allow these people to keep using the old code while technically being "up to date" and free of known vulnerabilities: https://reload4j.qos.ch/ https://reload4j.qos.ch/

TheKarateKid 3y ago

About a decade ago, the industry shifted from slow, carefully evaluated infrequent updates (except for security) to frequent, almost daily updates that are mandatory. I'd say this was pioneered by the Chromium team and proved to be beneficial. The rest of the industry followed. Now we're in a position where most projects update so quickly, that you don't really have a choice. If you need to update one component, there's a good chance it will require that many more as dependencies in your project will require an update to be compatible. The industry as a whole sacrificed stability and some aspects of security for faster advancements and features. Overall I'd say the net benefit is positive, but its times like these that remind us that perhaps we need to slow things down just a little and do a bit of a course correction to bring things into a better balance.

nilsherzig 3y ago

Wouldn't that just result in exploits written for old versions? A successful exploit for something that everyone is running might be worse, than a backdoor on blending edge systems. Everyone being on different versions results in something like a moving target

paulmd 3y ago

well, it's the bazaar vs the cathedral, isn't it? bazaar moves a lot faster. Everyone likes that part, except when it breaks things, and when they have to chase an upstream that's constantly churning, etc. but most people don't consider that a cathedral itself might have some engineering merit too. cathedrals are beautiful and polished and stable. I highly encourage people to try freeBSD sometime. Give ports a try (although the modern sense is that poudrie is better even if you want custom-built packages). See how nicely everything works. All the system options you need go into rc.conf (almost uniformly). Everything is documented and you can basically operate the system out of the FreeBSD Handbook documentation (it's not at all comparable to the "how to use a window or a menu" level intro stuff the linux provides). You can't do that when everything is furiously churning every release. everything just works, everything is just documented, it's an experience when you're coming from linux. and that forum post from 2007 on how to tweak a service script is probably still valid, because BSD hasn't had 3 different init systems over that timespan etc. just like "engineering is knowing how to build a bridge that barely doesn't fall over", engineering here is knowing what not to churn, and fitting your own work and functionality extensions into the existing patterns etc. like it doesn't have to be even "don't make a bigger change than you have to", you just have to present a stable userland and stable kernel interface and stable init/services interface. the fact that linux doesn't present a stable kernel interface is actually fairly sketchy/poor engineering, it doesn't have to be that way, a large subset of kernel interfaces probably should be stable.

Vicinity9635 3y ago

I'm kinda the opposite. Way too many times I've seen "upgrades" actively remove things I liked and add things I hate. I hold off on letting mobile apps update because they almost always get worse, not better.

hinkley 3y ago
I think it was Ted Neward who argued that libraries should come with a core and ancillary packages. Only the stuff really necessary and broadly popular goes into the base library, and all of the low use features should go into one or more ancillary libraries. I still think this is the way to go, but it does move some of the problem around in that you have to pick good cleave lines such that people aren’t forced to pull 5 libraries in for 5 features they need. You want one software domain to use two or three and another to use an overlapping set of three.

cryptonector 3y ago

That's a double-edged sword. What happens when you need to upgrade in order to get vulnerability fixes?

MySweetHubert 3y ago

The 2017 WannaCry ransomware attack would be a good counter example, the virus spread even though it was already fixed in an update from MS a bit more than a month before.

nickm12 3y ago

I don't believe this will result in conservatism around upgrading nor that it would be good if it did. The vast, vast majority of security vulnerabilities are due to regular bugs and once the details are known, they can be exploited. Not patching leaves you open to these exploits. Supply chain attacks are a real threat too, but being conservative about upgrading to improve your security posture is like saying "I'm worried someone with the key to my house might rob me, so I'm going to remove the lock from my door."

thegrim000 3y ago

This drove me crazy at previous companies where our build system dependencies were specified with wildcards so they would just auto pull in the latest version whenever you built the project. Not only are there security issues with that, as seen here, but it destroys your ability to have deterministic builds; every time you build the project it might be building against a different set of dependencies. You lose the ability to even compare builds against each other, because even though the builds are for the same exact project code, the dependency binaries might be different, and so the two builds could run differently.