7 ms·

I did a bunch of nodejs stuff at my last gig. These teams had the of practice keeping packages up to date. Drove me frikkin nuts. So much churn, chaos. Is this

I did a bunch of nodejs stuff at my last gig. These teams had the of practice keeping packages up to date. Drove me frikkin nuts. So much churn, chaos. Is this a JavaScript thing? Carried over from frontend development? Exasperated, I finally stopped advocating for locking everything down. Everyone treated me like I was crazy. (Reproducible builds?? Pfft!!) Happens with enterprisey Java, Sprint, Maven projects too. (I can't even comment on Python projects; I was just happy when I could get them to run.) What's going on? FOMO? Lock ~~Look~~ down dependencies. Only upgrade when absolutely necessary. Keep things simple to facilitate catching regression bugs and such. Oh well. I moved on. I don't miss it one bit.

oblio 5y ago
Java/Spring/Maven should have locked down dependencies by default. They have to go out of their way to not do that. Not that some people don't, anyway. Typos: > I stopped advocating for locking everything down. started? > Look down dependencies. lock?
- nvrspyx 5y ago
  I think the first typo is actually correct. They're saying that they stopped advocating for it because everyone treated them like they were crazy for doing so.
- specialist 5y ago
  Edited. Thanks for proofreading my rant. :)
- watermelon0 5y ago
  AFAIK Maven and Gradle don't have any built-in way of locking down dependencies (direct or transient), unless I missed something.
  - cesarb 5y ago
    In my experience, Maven always uses the exact version specified in your pom.xml (and the pom.xml of your dependencies, transitively), it never uses a newer version automatically. That is, the built-in way in Maven is locked down dependencies.
rtpg 5y ago
I mean you can have reproducible builds while being on the upgrade train. `package-lock.json` eixsts for a reason. And the tiny pains of upgrading packages over time mean that then you don't have to deal with gargantuan leaps when that one package has the thing you want and it requires updating 10 other packages because of dependencies. Node is a special horror because of absolute garbage like babel splitting itself into 100s of plugins and slowly killing the earth through useless HTTP requests instead of just packaging a single thing (also Jon Schlinkert wanting to up his package download counts by making a billion useless micropackages). But hey, you're choosing to use those packages. I think if you're using them, good to stay up to date. But you can always roll your own thing or just stay pinned. Just that stuff is still evolving in the JS world (since people still aren't super satisfied with the tooling). But more mature stuff is probably fine to stick to forever.
cube00 5y ago
Good luck when your security department starts running XRay and demands all your dependencies are at the latest versions all the time because every second release of a package is reported as vulnerable.
- cyberpunk 5y ago
  But.... We're right though? I mean there is some wiggle room, if some lib has a security hole in some functionality that's not actually being used, but it's all about risk. We gate deploys to prod over something like XRay or LifeCycle and the frustration it causes our devs is much cheaper than deploying something we know is insecure to prod and would cause us to fail an audit or something (let alone get pwned).
  - TechBro8615 5y ago
    There was a recent discussion on HN about `npm audit` and its overwhelming number of false positive vulnerabilities [0]. I can see a policy like this being frustrating in the case of `npm dependencies`. Is this something you deal with? [0] https://news.ycombinator.com/item?id=27761334 https://news.ycombinator.com/item?id=27761334
- specialist 5y ago
  Ya. That's a tough one. Philosophically, I totally support vetting. I don't know we'd make it useful. Pondering... One past team had a weekly formal "log bash" ritual. I LOVED it. That ops team kept track of everything. Any unexplained log entry had to be explained or eliminated. They kept track of explained entries on their end, so we wouldn't waste time rehashing stuff. I imagine their monitoring tools supported filters for squelching noisy explained log entries. Maybe the security team which owns XRay could do something similar.
wereHamster 5y ago
Everyone I know uses some form of lock file, and most of the modern programming languages support it. As for upgrading only when absolutely necessary, let's be honest, nothing is absolutely necessary. If the software is old, or slow, or buggy, well dear users you'll just have to deal with it. In my experience however, it's easier to keep dependencies relatively up to date all the time, and do the occasional change that goes along with each upgrade, than waiting five years until it's absolutely necessary, at which point upgrading will be a nightmare. I much rather spend each week 10 minutes reading through the short changelog of 5 dependencies to check that yes, that changes are simple enough that they can be merged without fear, and with the confidence that it's compatible with all the other up-to-date dependencies.
- specialist 5y ago
  How do you deal with regressions? For example, we once upgraded the redis client. One brief revision of the parser submodule had an apparent resource leak. (I can't imagine how...) Causing all of our services to ABEND after a few hours. Because everything is updated aggressively, and there's so many dependencies, we couldn't easily back out changes. -- FWIW, Gilt's "Test Into Production" strategy is the first and only sane regime I've heard for "Agile". Potentially reasonable successor to the era when teams did actual QA & Test. Sorry, I don't have a handy cite. I advocated for Test Into Prod. Alas, we didn't get very far before our ant farm got another good shake.
  - wereHamster 5y ago
    When I see a regression, I look at what was recently updated (in the past hours, days), and it's usually one of those packages. Because frequent updates tend to be independent, it's usually not difficult to revert that change (eg. if I update react and lodash at the same time, chances are really good that I can revert one of those changes independently without any issues)
    - actually_a_dog 5y ago
      This is the way. Also, you should try to update as few things as possible at once, and let your changes "soak" into production for a while before going in all "upgrade ALL the things!" Why? Well, sometimes things that fail take a while to actually start showing you symptoms. Sometimes, the failure itself takes a while to propagate, simply due to the number of servers you have. And, one of these days, cosmic rays, or some other random memory/disk error is going to flip a critical bit in some crucial file in that neato library you depend on. And, oh, the fouled up build is only going to go out to a subset of your machines. You'll be glad you didn't "upgrade ALL the things!" when any of these things ends up happening to you.
You-Are-Right 5y ago
This is not a solution.
wiremine 5y ago
Feels like this is a business opportunity for someone. Use case: 1. Upload a pre-reviewed package.json file. 2. The service monitors changes, and recommends updates. Recommendations might include security, bug, features, etc. It would check downstream dependencies, too. For production systems, the team might only care about security features. 3. Developer team can review recommendations, and download the new package.json. (There are lots of opportunities to improve this: direct integration with git, etc.) Anybody know if this sort of service exists? I know npm has _some_ of this. Maybe I'm just ignorant of how much of a solved problem this is?
- rank0 5y ago
  What you’re looking for is called SCA (software composition analysis). Best tool I’ve used so far in this domain is snyk.
- goodpoint 5y ago
  > Feels like this is a business opportunity for someone. This is what some Linux distributions do. Quality review, legal review, security backports.
- cilefen 5y ago
  We use Renovate for this.
Nursie 5y ago
> keeping packages up to date. It’s good security practice, especially for anything internet facing. Sure, you don’t have to do it obsessively, but if you let it stagnate you can have trouble updating things when critical vulnerabilities are found, and you have a huge job because multiple APIs have changed along the way.
_pmf_ 5y ago
They have no concept of ever needing to reproduce a build. And they are probably right if you do continuous development of some SaaS stuff.
- aranchelk 5y ago
  I’ve heard the argument that it’s not worth spending the time to get all aspects of your build environment into version control for 100% reproducible builds, but the saying “I shouldn’t be bothered to know what my upstream deps are” that’s pretty sad (hopefully rare). Even with the continuous deploy stuff, I think you’re giving too much benefit of the doubt. If an upstream non-pinned change brings down something critical (e.g. payments), you’ll revert the change in source, rebuild, release, the site is still broken. If you keep old artifacts you can push one of those, now you’re not really doing continuous deployment, and you won’t be again until you finish root cause analysis, and since you’ll never see the relevant change in your source code, it could take a while.
dheera 5y ago
I get notifications every other week about my NodeJS packages having security vulnerabilities, and so I upgrade. I'm not sure why NodeJS devs are so bad at security compared to say C++ devs. It's not like I'm getting asked to upgrade libstdc++ and eigen all the time ...
- ComputerGuru 5y ago
  I’m not going to get into barriers for entry and what kind of developer chooses which language, but C/C++ has always eschewed having unnecessary dependencies (originally because dependency management is hard, but that’s no longer the real reason - it’s just become baked into the culture). Even if you copy and paste hundreds of lines of code (or individual source files) the way you think about and treat them is very different from when they are external to your code base. You own (your copy of) them now, and at the very least, you familiarize yourself with their internals (and you wouldn’t copy and paste something that had its own dependencies). Actual (dynamically linked) dependencies don’t export objects with super brittle interfaces so much as they do expose very stable, carefully designed, and as minimal as possible boundaries for interfacing with their internals. Anything crossing API boundaries is considered leaving/entering the trusted domain, and lifespans of pointers and references in a non-garbage collected language guide/force you to eschew incorporating external libraries into your code globally and isolate the interaction (and therefore, the attack surface area) between your code and external code. Then there’s the type system. Despite having some of the weakest type systems of strongly types languages, C (to some extent) and C++ (especially modern C++) are still light years ahead of what even Typescript buys you because they don’t offer the same escape hatches or “fingers crossed this TS interface matches the JS object that we are binding it to, but two lines from here, no one will remember that this isn’t actually a native TS class and that this non-nullable field might actually be null because the compile-time type checking system can’t verify this even at runtime without manual user validation because the MS TS team refuses to transpile type validation because they insist on zero overhead even though JS is not a low level or systems language.” More than anything else however, developers of statically typed languages tend to fundamentally disagree with the idea of “move fast and break often” and will put off changes for years if they’re not at least 90% sure it’s the correct solution (and even when they end up wrong, it’s still far better than someone that says “who cares if it’s the or even just an actually correct solution, it’s still a solution and we can always revisit this later”).
fnord77 5y ago
on the other hand, if you don't keep your packages up to date, you can get so behind that it is nearly impossible to upgrade. especially bad if the older version you are on turns out to have vulns. Josh Bloch says to update your packages frequently and I agree.
Macha 5y ago
Backwards compatibility in the javascript world isn't great. If you stop updating for a couple of years, half your libraries have incompatible API changes. Then something like a node or UI framework update comes along and makes you update them all at once to work on the new version, and you're rewriting half your application just to move to a non-vulnerable version of your core dependency.
dec0dedab0de 5y ago
node locks down dependencies for you, not only the version, but it saves a hash too. The problem is that npm install will install the newest version allowed in your config, and re lock it. However if you run npm ci, it will only install what is locked, and fail if the hashes don't match. in python pipenv works the same way, pipenv sync will only install what is locked, and will check your hashes. I'm not sure about poetry.
Merad 5y ago
> Only upgrade when absolutely necessary. Gotta disagree with this part. If you're making a web app package updates really need to be done on a regular cadence, ideally every quarter, but twice a year at minimum IMO. In the .Net world at least it feels like most responsible open source maintainers make it relatively painless to upgrade between incremental major versions (e.g. v4 -> v5). If you put off upgrades until someone holds a gun to your head so that your dependencies are years out of date, you're much more likely to experience painful upgrades that require a lot of work.
actually_a_dog 5y ago
I agree. Locking things down is the way to go, when it comes to safety. The downside, however, is that, by design, you end up with packages that don't get upgraded regularly. That can cause problems down the road when you decide you do want to upgrade those packages. For instance, there might be breaking changes, because you're jumping major versions. Of course, breaking changes are always a problem, but, if you're not regularly upgrading stuff, your team will tend to build on/build around the functionality of the old version. That leads to some real fun come upgrade time. If, you're, say, 3 major versions behind the latest version, or whatever version you want to upgrade to that contains some Cool New Feature(tm) you really, really need, you might end up having to do this silly dance of upgrading major versions one at a time, in order to keep the complexity of the upgrade process as a whole under control. Oh, and, sometimes things get deprecated. That's always fun to deal with. So, TL;DR: Yes, pin versions! It's safer that way! Just be aware that, like most engineering decisions, there's a tradeoff here that saves you some pain now in exchange for some amount of a different kind of pain in the future.
paulddraper 5y ago
For quite a while now, the JS ecosystem has used lock files -- npm, yarn, pnpm. You have to got out of your way not to pin JS dependencies.
OJFord 5y ago
I tend to assume OSS contributors/maintainers are working generally to make things better, features I'm not using/don't need aside, they're fixing bugs (that I might not know I'm hitting/about to hit), patching security holes, etc. - so that's one for the 'pro-upgrading' column. Against that, sure, there might be something bad in the new release (maliciousness aside even, there might be a new bug!). But.. there might be in the old one I've pinned to as well? Assuming I'm not vetting everything (because how many places are, honestly) I have no reason to distrust the new version any more than my current one. Reproducible builds are an orthogonal issue? You can still keep your dependencies' versions updated with fully reproducible builds. Ours aren't, but we do pin to specific versions (requirements.txt & yarn.lock), and keep the former up to date with pyup (creates and tests a branch with the latest version) and up to date within semver minor versions just with yarn update (committed automatically since in theory it should be safe, had to revert only occasionally).

oblio 5y ago

Java/Spring/Maven should have locked down dependencies by default. They have to go out of their way to not do that. Not that some people don't, anyway. Typos: > I stopped advocating for locking everything down. started? > Look down dependencies. lock?

nvrspyx 5y ago
I think the first typo is actually correct. They're saying that they stopped advocating for it because everyone treated them like they were crazy for doing so.
specialist 5y ago
Edited. Thanks for proofreading my rant. :)
watermelon0 5y ago
AFAIK Maven and Gradle don't have any built-in way of locking down dependencies (direct or transient), unless I missed something.
- cesarb 5y ago
  In my experience, Maven always uses the exact version specified in your pom.xml (and the pom.xml of your dependencies, transitively), it never uses a newer version automatically. That is, the built-in way in Maven is locked down dependencies.

rtpg 5y ago

I mean you can have reproducible builds while being on the upgrade train. `package-lock.json` eixsts for a reason. And the tiny pains of upgrading packages over time mean that then you don't have to deal with gargantuan leaps when that one package has the thing you want and it requires updating 10 other packages because of dependencies. Node is a special horror because of absolute garbage like babel splitting itself into 100s of plugins and slowly killing the earth through useless HTTP requests instead of just packaging a single thing (also Jon Schlinkert wanting to up his package download counts by making a billion useless micropackages). But hey, you're choosing to use those packages. I think if you're using them, good to stay up to date. But you can always roll your own thing or just stay pinned. Just that stuff is still evolving in the JS world (since people still aren't super satisfied with the tooling). But more mature stuff is probably fine to stick to forever.

cube00 5y ago

Good luck when your security department starts running XRay and demands all your dependencies are at the latest versions all the time because every second release of a package is reported as vulnerable.

cyberpunk 5y ago
But.... We're right though? I mean there is some wiggle room, if some lib has a security hole in some functionality that's not actually being used, but it's all about risk. We gate deploys to prod over something like XRay or LifeCycle and the frustration it causes our devs is much cheaper than deploying something we know is insecure to prod and would cause us to fail an audit or something (let alone get pwned).
- TechBro8615 5y ago
  There was a recent discussion on HN about `npm audit` and its overwhelming number of false positive vulnerabilities [0]. I can see a policy like this being frustrating in the case of `npm dependencies`. Is this something you deal with? [0] https://news.ycombinator.com/item?id=27761334 https://news.ycombinator.com/item?id=27761334
specialist 5y ago
Ya. That's a tough one. Philosophically, I totally support vetting. I don't know we'd make it useful. Pondering... One past team had a weekly formal "log bash" ritual. I LOVED it. That ops team kept track of everything. Any unexplained log entry had to be explained or eliminated. They kept track of explained entries on their end, so we wouldn't waste time rehashing stuff. I imagine their monitoring tools supported filters for squelching noisy explained log entries. Maybe the security team which owns XRay could do something similar.

wereHamster 5y ago

Everyone I know uses some form of lock file, and most of the modern programming languages support it. As for upgrading only when absolutely necessary, let's be honest, nothing is absolutely necessary. If the software is old, or slow, or buggy, well dear users you'll just have to deal with it. In my experience however, it's easier to keep dependencies relatively up to date all the time, and do the occasional change that goes along with each upgrade, than waiting five years until it's absolutely necessary, at which point upgrading will be a nightmare. I much rather spend each week 10 minutes reading through the short changelog of 5 dependencies to check that yes, that changes are simple enough that they can be merged without fear, and with the confidence that it's compatible with all the other up-to-date dependencies.

specialist 5y ago
How do you deal with regressions? For example, we once upgraded the redis client. One brief revision of the parser submodule had an apparent resource leak. (I can't imagine how...) Causing all of our services to ABEND after a few hours. Because everything is updated aggressively, and there's so many dependencies, we couldn't easily back out changes. -- FWIW, Gilt's "Test Into Production" strategy is the first and only sane regime I've heard for "Agile". Potentially reasonable successor to the era when teams did actual QA & Test. Sorry, I don't have a handy cite. I advocated for Test Into Prod. Alas, we didn't get very far before our ant farm got another good shake.
- wereHamster 5y ago
  When I see a regression, I look at what was recently updated (in the past hours, days), and it's usually one of those packages. Because frequent updates tend to be independent, it's usually not difficult to revert that change (eg. if I update react and lodash at the same time, chances are really good that I can revert one of those changes independently without any issues)
  - actually_a_dog 5y ago
    This is the way. Also, you should try to update as few things as possible at once, and let your changes "soak" into production for a while before going in all "upgrade ALL the things!" Why? Well, sometimes things that fail take a while to actually start showing you symptoms. Sometimes, the failure itself takes a while to propagate, simply due to the number of servers you have. And, one of these days, cosmic rays, or some other random memory/disk error is going to flip a critical bit in some crucial file in that neato library you depend on. And, oh, the fouled up build is only going to go out to a subset of your machines. You'll be glad you didn't "upgrade ALL the things!" when any of these things ends up happening to you.

You-Are-Right 5y ago

This is not a solution.

wiremine 5y ago

Feels like this is a business opportunity for someone. Use case: 1. Upload a pre-reviewed package.json file. 2. The service monitors changes, and recommends updates. Recommendations might include security, bug, features, etc. It would check downstream dependencies, too. For production systems, the team might only care about security features. 3. Developer team can review recommendations, and download the new package.json. (There are lots of opportunities to improve this: direct integration with git, etc.) Anybody know if this sort of service exists? I know npm has _some_ of this. Maybe I'm just ignorant of how much of a solved problem this is?

rank0 5y ago
What you’re looking for is called SCA (software composition analysis). Best tool I’ve used so far in this domain is snyk.
goodpoint 5y ago
> Feels like this is a business opportunity for someone. This is what some Linux distributions do. Quality review, legal review, security backports.
cilefen 5y ago
We use Renovate for this.

Nursie 5y ago

> keeping packages up to date. It’s good security practice, especially for anything internet facing. Sure, you don’t have to do it obsessively, but if you let it stagnate you can have trouble updating things when critical vulnerabilities are found, and you have a huge job because multiple APIs have changed along the way.

_pmf_ 5y ago

They have no concept of ever needing to reproduce a build. And they are probably right if you do continuous development of some SaaS stuff.

aranchelk 5y ago
I’ve heard the argument that it’s not worth spending the time to get all aspects of your build environment into version control for 100% reproducible builds, but the saying “I shouldn’t be bothered to know what my upstream deps are” that’s pretty sad (hopefully rare). Even with the continuous deploy stuff, I think you’re giving too much benefit of the doubt. If an upstream non-pinned change brings down something critical (e.g. payments), you’ll revert the change in source, rebuild, release, the site is still broken. If you keep old artifacts you can push one of those, now you’re not really doing continuous deployment, and you won’t be again until you finish root cause analysis, and since you’ll never see the relevant change in your source code, it could take a while.

dheera 5y ago

I get notifications every other week about my NodeJS packages having security vulnerabilities, and so I upgrade. I'm not sure why NodeJS devs are so bad at security compared to say C++ devs. It's not like I'm getting asked to upgrade libstdc++ and eigen all the time ...

ComputerGuru 5y ago
I’m not going to get into barriers for entry and what kind of developer chooses which language, but C/C++ has always eschewed having unnecessary dependencies (originally because dependency management is hard, but that’s no longer the real reason - it’s just become baked into the culture). Even if you copy and paste hundreds of lines of code (or individual source files) the way you think about and treat them is very different from when they are external to your code base. You own (your copy of) them now, and at the very least, you familiarize yourself with their internals (and you wouldn’t copy and paste something that had its own dependencies). Actual (dynamically linked) dependencies don’t export objects with super brittle interfaces so much as they do expose very stable, carefully designed, and as minimal as possible boundaries for interfacing with their internals. Anything crossing API boundaries is considered leaving/entering the trusted domain, and lifespans of pointers and references in a non-garbage collected language guide/force you to eschew incorporating external libraries into your code globally and isolate the interaction (and therefore, the attack surface area) between your code and external code. Then there’s the type system. Despite having some of the weakest type systems of strongly types languages, C (to some extent) and C++ (especially modern C++) are still light years ahead of what even Typescript buys you because they don’t offer the same escape hatches or “fingers crossed this TS interface matches the JS object that we are binding it to, but two lines from here, no one will remember that this isn’t actually a native TS class and that this non-nullable field might actually be null because the compile-time type checking system can’t verify this even at runtime without manual user validation because the MS TS team refuses to transpile type validation because they insist on zero overhead even though JS is not a low level or systems language.” More than anything else however, developers of statically typed languages tend to fundamentally disagree with the idea of “move fast and break often” and will put off changes for years if they’re not at least 90% sure it’s the correct solution (and even when they end up wrong, it’s still far better than someone that says “who cares if it’s the or even just an actually correct solution, it’s still a solution and we can always revisit this later”).

fnord77 5y ago

on the other hand, if you don't keep your packages up to date, you can get so behind that it is nearly impossible to upgrade. especially bad if the older version you are on turns out to have vulns. Josh Bloch says to update your packages frequently and I agree.

Macha 5y ago

Backwards compatibility in the javascript world isn't great. If you stop updating for a couple of years, half your libraries have incompatible API changes. Then something like a node or UI framework update comes along and makes you update them all at once to work on the new version, and you're rewriting half your application just to move to a non-vulnerable version of your core dependency.

dec0dedab0de 5y ago

node locks down dependencies for you, not only the version, but it saves a hash too. The problem is that npm install will install the newest version allowed in your config, and re lock it. However if you run npm ci, it will only install what is locked, and fail if the hashes don't match. in python pipenv works the same way, pipenv sync will only install what is locked, and will check your hashes. I'm not sure about poetry.

Merad 5y ago

> Only upgrade when absolutely necessary. Gotta disagree with this part. If you're making a web app package updates really need to be done on a regular cadence, ideally every quarter, but twice a year at minimum IMO. In the .Net world at least it feels like most responsible open source maintainers make it relatively painless to upgrade between incremental major versions (e.g. v4 -> v5). If you put off upgrades until someone holds a gun to your head so that your dependencies are years out of date, you're much more likely to experience painful upgrades that require a lot of work.

actually_a_dog 5y ago

I agree. Locking things down is the way to go, when it comes to safety. The downside, however, is that, by design, you end up with packages that don't get upgraded regularly. That can cause problems down the road when you decide you do want to upgrade those packages. For instance, there might be breaking changes, because you're jumping major versions. Of course, breaking changes are always a problem, but, if you're not regularly upgrading stuff, your team will tend to build on/build around the functionality of the old version. That leads to some real fun come upgrade time. If, you're, say, 3 major versions behind the latest version, or whatever version you want to upgrade to that contains some Cool New Feature(tm) you really, really need, you might end up having to do this silly dance of upgrading major versions one at a time, in order to keep the complexity of the upgrade process as a whole under control. Oh, and, sometimes things get deprecated. That's always fun to deal with. So, TL;DR: Yes, pin versions! It's safer that way! Just be aware that, like most engineering decisions, there's a tradeoff here that saves you some pain now in exchange for some amount of a different kind of pain in the future.

paulddraper 5y ago

For quite a while now, the JS ecosystem has used lock files -- npm, yarn, pnpm. You have to got out of your way not to pin JS dependencies.

OJFord 5y ago

I tend to assume OSS contributors/maintainers are working generally to make things better, features I'm not using/don't need aside, they're fixing bugs (that I might not know I'm hitting/about to hit), patching security holes, etc. - so that's one for the 'pro-upgrading' column. Against that, sure, there might be something bad in the new release (maliciousness aside even, there might be a new bug!). But.. there might be in the old one I've pinned to as well? Assuming I'm not vetting everything (because how many places are, honestly) I have no reason to distrust the new version any more than my current one. Reproducible builds are an orthogonal issue? You can still keep your dependencies' versions updated with fully reproducible builds. Ours aren't, but we do pin to specific versions (requirements.txt & yarn.lock), and keep the former up to date with pyup (creates and tests a branch with the latest version) and up to date within semver minor versions just with yarn update (committed automatically since in theory it should be safe, had to revert only occasionally).