7 ms·

There's a lesson to be drawn here about dependency hygiene. On one hand, code-reuse is a good thing, and incorporating other projects by reference is a way to m

There's a lesson to be drawn here about dependency hygiene. On one hand, code-reuse is a good thing, and incorporating other projects by reference is a way to make it happen. On the other hand, each dependency you add creates a little bit of risk: that the package will be updated in a way that breaks your application, or the package maintainer will go rogue, or the package will get hijacked. Most programming-language communities manage the code-reuse/dependency-hygiene tradeoff by concentrating code into a relatively small number of large libraries, and when people want a function or two that isn't in one of the libraries they're using, they incorporate it by copy-paste. In the Javascript/NPM world, on the other hand, I see a lot of projects with dependency references to huge numbers of tiny dependencies. Today, we're seeing one of the reasons why that's a liability. Most people will take away lessons about code signing and signatures, and presumably the NPM software is going to be improved in that regard. But the other lesson is that projects should have fewer dependencies. Using a library may be as simple as adding one line to a package configuration file, but using a library properly requires substantial due diligence on the library, its license, its bug tracker and its author.

draw_down 11y ago
No, no. This is about whether people should be able to unpublish from the registry, or how that should be handled since it's problematic in any case.
- Xiol 11y ago
  God, I wish I could downvote you. As it is, this is the first time I've logged in for about 12 months, just to berate you.
- ntpeters 11y ago
  Honestly, I think you're both correct. It's definitely dangerous to allow packages to be unpublished, but it can potentially be just as dangerous for people to blindly include a library without looking into a bit first. First lesson learned is of course in regard to how package managers such as NPM should handle scenarios like this. However, I would also hope this might make some people take a harder look at their dependencies to see if everything they are referencing is both truly needed and trustworthy.
- belltaco 11y ago
  Your comment reminds me of what SourceForge did. >22 Apr 2015: By this date, SourceForge Open Mirror Directory has expanded to take over popular former SourceForge projects which left the site, such as VLC and GIMP for Windows. Some projects have malware embedded despite earlier assurances that the adware system would be purely opt-in. >16 May 2015: GIMP developer asks SourceForge to remove gimp-win from the site. They don’t. SourceForge later claims they didn’t receive this message.
- apalmer 11y ago
  you are part of the problem here. Many folks are refusing to even consider the fundamental issue because it is different from the mindset they are used to. Yes this particular episode is related to the unpublishing results... however the underlying issue is not only about unpublishing results. The underlying issue is: number 1, Having thousands of dependencies is a software engineering trade off, their is a upside and a downside of that trade off. Having Thousands of dependencies is fundamentally more 'risky' than have a few. Period. End Of Story. It is ok to accept the trade off based on what you are optimizing for. It is not ok to refuse to accept the fact you are making a tradeoff. number 2, is you should not be running production builds or builds for important components based on dependencies on the naked internet. This is also an engineering tradeoff but is firmly in the camp of what is considered 'bad practice'. This is fundamentally the same exact underlying reason why we have build servers instead of just builds from developer work stations. Developer workstations are not controlled environments, which means the builds are not guaranteed repeatable. A build server that depends on the state of the naked internet is not a controlled environment. It blows up today because of unpublishing, it will blow up tomorrow because of something else.
  - Wintamute 11y ago
    I agree with what you're saying. But I think the "grand experiment" of npm and modern javascript development is to go "all in" on the "lots of small modules" approach. Having many small dependencies is, and always has been, a trade off. The question is how far can we get optimising away from the pitfalls, while still retaining the benefits? I think it's a worthy exercise, since the prize is no less than efficiently distributing coding effort across the globe. Problems like we've seen today are a double edged sword. On the one hand they're troubling, but it'll focus the community on the problems, and hopefully npm will make a few changes too. > A build server that depends on the state of the naked internet Whether npm based or not, don't the vast majority of build servers, for whatever language or platform, rely on the state of the naked internet? Whether that's downloading binaries, compiling from source, or whatever?
    - clappski 11y ago
      Your build servers shouldn't really be exposed to the public internet, best practise would be to get all dependencies/binaries of a local network.
kbenson 11y ago
> Most programming-language communities manage the code-reuse/dependency-hygiene tradeoff by concentrating code into a relatively small number of large libraries, and when people want a function or two that isn't in one of the libraries they're using, they incorporate it by copy-paste. I don't think it's necessarily "most programming languages". I know Perl, and probably Python and Ruby, have very large ecosystems of small modules. I suspect it's more along the lines of static vs dynamic, or compiled vs interpreted. > On the other hand, each dependency you add creates a little bit of risk: that the package will be updated in a way that breaks your application, or the package maintainer will go rogue, or the package will get hijacked. Those specific risks are not necessarily from including a module, but are from subscribing to a module. If you include a specific version of a module and can confirm it has not changed to a degree that satisfies you (whether that's keeping a local copy to build from, or trusting the distribution system to be immutable and pegging a specific version to install), then your risk is fairly well defined. It's not zero, but it is limited to the problems the module existed with at the point you reviewed and included it. Subscribing to a module, that is having the system download and build "the latest and greatest of whatever thing you call $foo" is not very safe, and if the build mechanism can execute arbitrary code and is automated, is insanity.
- wereHamster 11y ago
  > If you include a specific version of a module That only works if the central registry is append-only. But as we've seen packages can be removed from npmjs, which caused these problems.
  - kbenson 11y ago
    The scope of my comment is obviously beyond just NPM, and I addressed that specific complaint, so I'm not sure how this comment follows mine.
- msbarnett 11y ago
  > I don't think it's necessarily "most programming languages". I know Perl, and probably Python and Ruby, have very large ecosystems of small modules. I suspect it's more along the lines of static vs dynamic, or compiled vs interpreted. The Perl, Python, and Ruby communities might have a lot of "small-ish" modules, compared to something like Boost in C++, but the Node.js community has really taken it to a new extreme with an explosion of "one-liner" modules like the absurd user-home https://github.com/sindresorhus/user-home https://github.com/sindresorhus/user-home The rallying cry of this movement is "Modules are Cheap in Node.js!". Incidents like this help demonstrate that explosive growth of your dependency graph isn't ever cheap, in any language.
pc86 11y ago
> On the other hand, each dependency you add creates a little bit of risk: that the package will be updated in a way that breaks your application, or the package maintainer will go rogue, or the package will get hijacked. If your development environment is such that you can't test these things before going into production, and that production updates its packages on its own, then you have much bigger (and harder to fix) issues that dependency hygiene.
- nyan4 11y ago
  > before going into production How about 3 years after a release? 5, 8, 10?
  - philipov 11y ago
    If you haven't upgraded your packages in 10 years, you've got nothing to worry about ;)
- zenlikethat 11y ago
  The risk is much bigger than simple lifecycle testing will account for. Upstream micro libraries could easily disappear or have bugs, leaving you spending more time trying to workaround than if you had just written the darn thing yourself in the first place. Or, you suddenly have a new requirement that doesn't play well with the lib. Programmers loves dependencies because it lets them pass the buck, but each and every 3rd party module is a potential time bomb waiting to happen.
- michaelmcmillan 11y ago
  How can a test detect a hijacked dependency that acts exactly like the original, but with a backdoor?
nyan4 11y ago
This is a decade old lesson, well understood by few developers that want their code to be around for a while. The HN "web developers" are a different crowd.
sotojuan 11y ago
What about using "trustable" dependencies? For small utilities like left pad, you could use lodash. You can import just the function, but still get the benefit of knowing that its lodash so it won't go down, it'll be fast, and well tested.
- AgentME 11y ago
  Left-pad was a trustable, fast, well-tested package too, up until NPM let it disappear.
  - esailija 11y ago
    How is it fast? (obviously it was not trustable or well-tested) It is the slowest possible way to implement string padding without any optimization effort whatsoever. This is what I hate about npm, every module claims to be fast even though they have made no effort to be fast.
    - goldbrick 11y ago
      It's fast 'cus it uses libuv, duh!
- PeCaN 11y ago
  This is what a lot of non-JavaScript languages do. I suspect JavaScript ended up this way because people wanted to avoid downloading anything they absolutely didn't need in the browser. Now packagers can automatically remove unused functions (WebPack 2 does this IIRC, and I think require.js did ages ago) so it's no longer a concern but somehow NPM still ended up with a million tiny libraries. Thank god Underscore/lodash didn't use the approach of one-module-per-function.
  - Macha 11y ago
    one-module-per-function is exactly the approach Lodash uses and the Lodash devs tout is as a big advantage over underscore. The difference is they also provide the rollup bundle with all of Lodash. Which is what most users actually use.
igorgue 11y ago
Isn't it a bigger lesson to get that Open Source project are for the people and not the interest of a corporation?
maxthegeek1 11y ago
The difference here is that many people in the npm world are working on the web. Minimizing application size is of absolute importance on the web. If we copy and past snippets of code between packages, we lose out on potential code de-duplication from dependency flattening. If you're really worried about your dependencies disappearing/changing beneath you just check them into git.
- peterhadlaw 11y ago
  This is a great comment, especially the advice to use git. Sometimes you may have dependencies, and that's okay. Keep track of them, see what changes in your own capacity. Also an added benefit is if the unicorn dependency distribution system goes bust, you still have local copies of everything.
  - lewisl9029 11y ago
    I tried checking in all dependencies for one of my projects a while ago, and my git history quickly became ridiculously bloated as I added/upgraded/replaced more and more dependencies as time went on. For those who go this route, any advice on what you can do to keep your version control history from blowing up?
    - wlesieutre 11y ago
      Git submodules maybe? https://git-scm.com/docs/git-submodule https://git-scm.com/docs/git-submodule
      - lewisl9029 11y ago
        Ahh... That must be what I was missing. So instead of checking in your dependencies into your app repo directly, you turn your dependencies folders into submodules to keep their histories separate from your app repo's history. Makes sense! I'll be sure to give that a try on my next project.
        11y ago
deleted 11y ago
[deleted]

draw_down 11y ago

No, no. This is about whether people should be able to unpublish from the registry, or how that should be handled since it's problematic in any case.

Xiol 11y ago
God, I wish I could downvote you. As it is, this is the first time I've logged in for about 12 months, just to berate you.
ntpeters 11y ago
Honestly, I think you're both correct. It's definitely dangerous to allow packages to be unpublished, but it can potentially be just as dangerous for people to blindly include a library without looking into a bit first. First lesson learned is of course in regard to how package managers such as NPM should handle scenarios like this. However, I would also hope this might make some people take a harder look at their dependencies to see if everything they are referencing is both truly needed and trustworthy.
belltaco 11y ago
Your comment reminds me of what SourceForge did. >22 Apr 2015: By this date, SourceForge Open Mirror Directory has expanded to take over popular former SourceForge projects which left the site, such as VLC and GIMP for Windows. Some projects have malware embedded despite earlier assurances that the adware system would be purely opt-in. >16 May 2015: GIMP developer asks SourceForge to remove gimp-win from the site. They don’t. SourceForge later claims they didn’t receive this message.
apalmer 11y ago
you are part of the problem here. Many folks are refusing to even consider the fundamental issue because it is different from the mindset they are used to. Yes this particular episode is related to the unpublishing results... however the underlying issue is not only about unpublishing results. The underlying issue is: number 1, Having thousands of dependencies is a software engineering trade off, their is a upside and a downside of that trade off. Having Thousands of dependencies is fundamentally more 'risky' than have a few. Period. End Of Story. It is ok to accept the trade off based on what you are optimizing for. It is not ok to refuse to accept the fact you are making a tradeoff. number 2, is you should not be running production builds or builds for important components based on dependencies on the naked internet. This is also an engineering tradeoff but is firmly in the camp of what is considered 'bad practice'. This is fundamentally the same exact underlying reason why we have build servers instead of just builds from developer work stations. Developer workstations are not controlled environments, which means the builds are not guaranteed repeatable. A build server that depends on the state of the naked internet is not a controlled environment. It blows up today because of unpublishing, it will blow up tomorrow because of something else.
- Wintamute 11y ago
  I agree with what you're saying. But I think the "grand experiment" of npm and modern javascript development is to go "all in" on the "lots of small modules" approach. Having many small dependencies is, and always has been, a trade off. The question is how far can we get optimising away from the pitfalls, while still retaining the benefits? I think it's a worthy exercise, since the prize is no less than efficiently distributing coding effort across the globe. Problems like we've seen today are a double edged sword. On the one hand they're troubling, but it'll focus the community on the problems, and hopefully npm will make a few changes too. > A build server that depends on the state of the naked internet Whether npm based or not, don't the vast majority of build servers, for whatever language or platform, rely on the state of the naked internet? Whether that's downloading binaries, compiling from source, or whatever?
  - clappski 11y ago
    Your build servers shouldn't really be exposed to the public internet, best practise would be to get all dependencies/binaries of a local network.

kbenson 11y ago

> Most programming-language communities manage the code-reuse/dependency-hygiene tradeoff by concentrating code into a relatively small number of large libraries, and when people want a function or two that isn't in one of the libraries they're using, they incorporate it by copy-paste. I don't think it's necessarily "most programming languages". I know Perl, and probably Python and Ruby, have very large ecosystems of small modules. I suspect it's more along the lines of static vs dynamic, or compiled vs interpreted. > On the other hand, each dependency you add creates a little bit of risk: that the package will be updated in a way that breaks your application, or the package maintainer will go rogue, or the package will get hijacked. Those specific risks are not necessarily from including a module, but are from subscribing to a module. If you include a specific version of a module and can confirm it has not changed to a degree that satisfies you (whether that's keeping a local copy to build from, or trusting the distribution system to be immutable and pegging a specific version to install), then your risk is fairly well defined. It's not zero, but it is limited to the problems the module existed with at the point you reviewed and included it. Subscribing to a module, that is having the system download and build "the latest and greatest of whatever thing you call $foo" is not very safe, and if the build mechanism can execute arbitrary code and is automated, is insanity.

wereHamster 11y ago
> If you include a specific version of a module That only works if the central registry is append-only. But as we've seen packages can be removed from npmjs, which caused these problems.
- kbenson 11y ago
  The scope of my comment is obviously beyond just NPM, and I addressed that specific complaint, so I'm not sure how this comment follows mine.
msbarnett 11y ago
> I don't think it's necessarily "most programming languages". I know Perl, and probably Python and Ruby, have very large ecosystems of small modules. I suspect it's more along the lines of static vs dynamic, or compiled vs interpreted. The Perl, Python, and Ruby communities might have a lot of "small-ish" modules, compared to something like Boost in C++, but the Node.js community has really taken it to a new extreme with an explosion of "one-liner" modules like the absurd user-home https://github.com/sindresorhus/user-home https://github.com/sindresorhus/user-home The rallying cry of this movement is "Modules are Cheap in Node.js!". Incidents like this help demonstrate that explosive growth of your dependency graph isn't ever cheap, in any language.

pc86 11y ago

> On the other hand, each dependency you add creates a little bit of risk: that the package will be updated in a way that breaks your application, or the package maintainer will go rogue, or the package will get hijacked. If your development environment is such that you can't test these things before going into production, and that production updates its packages on its own, then you have much bigger (and harder to fix) issues that dependency hygiene.

nyan4 11y ago
> before going into production How about 3 years after a release? 5, 8, 10?
- philipov 11y ago
  If you haven't upgraded your packages in 10 years, you've got nothing to worry about ;)
zenlikethat 11y ago
The risk is much bigger than simple lifecycle testing will account for. Upstream micro libraries could easily disappear or have bugs, leaving you spending more time trying to workaround than if you had just written the darn thing yourself in the first place. Or, you suddenly have a new requirement that doesn't play well with the lib. Programmers loves dependencies because it lets them pass the buck, but each and every 3rd party module is a potential time bomb waiting to happen.
michaelmcmillan 11y ago
How can a test detect a hijacked dependency that acts exactly like the original, but with a backdoor?

nyan4 11y ago

This is a decade old lesson, well understood by few developers that want their code to be around for a while. The HN "web developers" are a different crowd.

sotojuan 11y ago

What about using "trustable" dependencies? For small utilities like left pad, you could use lodash. You can import just the function, but still get the benefit of knowing that its lodash so it won't go down, it'll be fast, and well tested.

AgentME 11y ago
Left-pad was a trustable, fast, well-tested package too, up until NPM let it disappear.
- esailija 11y ago
  How is it fast? (obviously it was not trustable or well-tested) It is the slowest possible way to implement string padding without any optimization effort whatsoever. This is what I hate about npm, every module claims to be fast even though they have made no effort to be fast.
  - goldbrick 11y ago
    It's fast 'cus it uses libuv, duh!
PeCaN 11y ago
This is what a lot of non-JavaScript languages do. I suspect JavaScript ended up this way because people wanted to avoid downloading anything they absolutely didn't need in the browser. Now packagers can automatically remove unused functions (WebPack 2 does this IIRC, and I think require.js did ages ago) so it's no longer a concern but somehow NPM still ended up with a million tiny libraries. Thank god Underscore/lodash didn't use the approach of one-module-per-function.
- Macha 11y ago
  one-module-per-function is exactly the approach Lodash uses and the Lodash devs tout is as a big advantage over underscore. The difference is they also provide the rollup bundle with all of Lodash. Which is what most users actually use.

igorgue 11y ago

Isn't it a bigger lesson to get that Open Source project are for the people and not the interest of a corporation?

maxthegeek1 11y ago

The difference here is that many people in the npm world are working on the web. Minimizing application size is of absolute importance on the web. If we copy and past snippets of code between packages, we lose out on potential code de-duplication from dependency flattening. If you're really worried about your dependencies disappearing/changing beneath you just check them into git.

peterhadlaw 11y ago
This is a great comment, especially the advice to use git. Sometimes you may have dependencies, and that's okay. Keep track of them, see what changes in your own capacity. Also an added benefit is if the unicorn dependency distribution system goes bust, you still have local copies of everything.
- lewisl9029 11y ago
  I tried checking in all dependencies for one of my projects a while ago, and my git history quickly became ridiculously bloated as I added/upgraded/replaced more and more dependencies as time went on. For those who go this route, any advice on what you can do to keep your version control history from blowing up?
  - wlesieutre 11y ago
    Git submodules maybe? https://git-scm.com/docs/git-submodule https://git-scm.com/docs/git-submodule
    - lewisl9029 11y ago
      Ahh... That must be what I was missing. So instead of checking in your dependencies into your app repo directly, you turn your dependencies folders into submodules to keep their histories separate from your app repo's history. Makes sense! I'll be sure to give that a try on my next project.
      - 11y ago

deleted 11y ago

[deleted]