9 ms·

The Node.js Ecosystem Is Chaotic and Insecure

118 points by unicornode 8y ago

pavlov 8y ago
In my view the core problem is the complete lack of standards and interfaces around what constitutes a “package”, “module”, “library” or even “framework”. (It’s an industry-wide problem, not just JavaScript.) In npm’s single global namespace, “is-even” is a package. So is the Firebase SDK, even though the former is a single line of code and the latter is around 70 MB. But a package can also be something that’s not even JavaScript code: maybe a CSS “framework”, or that infamous dictionary text that gets replicated a dozen times in any large npm-dependent project because some popular package uses the text in its tests. If hardware parts were distributed like this, there would be no standard fittings, nothing would be categorized. A hardware store would just be an enormous pile of jumbled parts and tools. “Oh, you need a bolt? No problem, John will jump into the pile with scuba gear and bring you some packages that look suitable. It’s going to take a few days, hopefully we’ll find something that fits.”
- treve 8y ago
  What are you looking for exactly? A category system? The argument seems kind of weird. A bit similar to how a .zip file might contain just a few things, but also a lot of things. I think it's kind of a good thing that the same delivery mechanism can be used for various things, but I'm guessing you have a problem with discoverability?
  - pavlov 8y ago
    The problem affects not just discoverability of new dependencies, but also development and maintenance of settled dependencies. When you see a line like this in package.json... "dependencies": { "foobarfrobblizer": "^0.9.31", There's no simple way to determine what foobarfrobblizer does, where it's actually used in the app, what side effects it may perform when called, what dependencies it pulls, why it was included, and whether it could be replaced with something else that offers a similar API. For a single package, I can figure all this out manually. But projects these days are seriously dependency-binged. If I use "create-react-app" to make a Hello World web app, it gives me 150 MB of stuff in node_modules. I'm never going to look at even 1% of those packages to see what's actually in there. I got started in programming when a dependency meant either a DLL file provided by someone else, or a static library that you manually link into your executable. Taking on a new dependency was a big decision: do I really want to rely on this DLL file being there for me, or having to carry this separately built binary blob in my project? I still get that same dread when I have to add an npm package to a project, but it doesn't seem like most people feel that way. On top of those DLL files 25 years ago, there were systems like Microsoft COM that let you use them interchangeably through standard interfaces. You could make a plugin UI component that works seamlessly in Visual Basic by simply conforming to a COM interface. It even worked from multiple languages: that UI component could be written in C, C++ or Object Pascal if you wanted. The web development ecosystem has a long way to go before it matches the usability of 1995 desktop development — and that's a painfully low bar.
    - askmike 8y ago
      > Taking on a new dependency was a big decision: do I really want to rely on this DLL file being there for me, or having to carry this separately built binary blob in my project? I'd argue that the NPM ecosystem (although not perfect, hence OP) is a lot more evolved specifically because of how easy it is to include a wide range of dependencies. The speed with which you can write software is an awful lot higher, and (if done properly) you can rely on safe dependencies that are tested and used by everyone else. Why reinvent the wheel in every project?
      - raarts 8y ago
        > you can rely on This is exactly the problem. Given the current ecosystem, you can't and shouldn't. You are just proving GP's point.
    - krapp 8y ago
      >There's no simple way to determine what foobarfrobblizer does, where it's actually used in the app, what side effects it may perform when called, what dependencies it pulls, why it was included, and whether it could be replaced with something else that offers a similar API. Isn't this an application issue? The manifest for each project should tell you that, it should be possible to build the dependency tree as a preview and see all of the metadata in aggregate.
sbr464 8y ago
Click bait, move along
- movedx 8y ago
  Are you are JavaScript developer, out of interest?
  - sbr464 8y ago
    yes
    - movedx 8y ago
      I'm not the author of the article, but could you expand on why it's click bait? Personally, I think NPM is a crazy, mental concept. As a Go developer (so mostly server-side work) I can't fathom the amount of deps that are pulled from NPM when I'm working with or watching a JS code base being used. I assume server-side NodeJS code bases are the same their front end cousins in terms of deps pulled(?)
      - sbr464 8y ago
        Hi, appreciate the followup, apologies for the previous short comment. I feel like it's clickbait, not worth reading, for these points: 1. Even if the argument is valid, the points made in this article have been expressed, shared, (whined about?) probably 9+ times in my own reading. It's really time to move on. 2. There is nothing inherently wrong with using small/simple modules. Once you start to reuse code in many projects, the value becomes clear. Some are juicy and complex, some are eye-bleeding simple. It's better than copying and pasting code around folders. Use org modules/private modules and the problem is solved. I absolutely believe that npm is ripe for *potential abuse, although I have yet to have any issues. No one has time for Breitbart-esque fearmongering, or the regurgitation of stale points while an ample selection of real world problems exist to be solved. Jeez that reads harsh. Apologies, it's late.
        deleted 8y ago
wyqydsyq 8y ago
yet-another-article-talking-about-the-left-pad-incident-and-fragile-dependency-chains-in-general-with-a-sensationalized-clickbait-title
- movedx 8y ago
  Are you are JavaScript developer, out of interest?
- deleted 8y ago
  [deleted]
baybal2 8y ago
The "modern" JS ecosystem: 1. Over-reliance on tooling. JS was made as a scripting language, if you can't use it as such, its purpose is lost. And that tooling works against betterment of the ecosystem - lib authors rely on tooling to fix issues of the platform, rather than seeking a definite solution. 2. Any serious JS project is a long term maintenance burden. The prevalence of "single line libraries" makes future-proofing of the code impossible. You don't only have to version-freeze your immediate dependencies, but dependencies of your dependencies. You will have to manage their version manually, it all becomes your own problem. 3. Authors of JS VMs swear on their firstborns that JS performance has been greatly improved, yet, it sucks. For practical purposes, JS code is barely as quick as perl or python. 4. Too much evals in 3rd party code everywhere. The only way you can secure your front-end is to ensure no third party code policy on your page. When it comes to the back-end, things become more grave. On the back-end, you can say that there is no universal solution. 5. Zoo of standards on the front-end, browser APIs changing on a whim. 6. Zoo of sublanguages: Coffeescript, LiveScript ClojureScript, MoonScript, AtScript, TypeScript. Having to mix anymuch of them in a single project is, you can guess, a supernightmare. Yet, I don't completely hate people who want to move JS forward. TC39 may well be too heavy on being staffed by theoreticians, but they have right priorities: standardization, having whatever going into the standard being highly dry before even approaching the pre-standard status, enforcing a semblance of synchronicity to the ecosystem. What JS needs in my opinion: 1. HARD VERSIONING 2. Compliance certification through testing with a hard version 3. An even harder enforced decoupling of the paper standard and implementation. While the majority of modern JS implementations are VM based, this should not make the standard to accommodate design decisions dictated by a VM implementation. 4. Readiness to do breaking changes in between major hard versions, and no reservations for backward compatibility. 5. Many browser APIs are de-facto parts of the standard library as such. They need to be managed as a part of the standard.
- movedx 8y ago
  Genuine question: does transpiling from Go, for example, make things any better? I've often wondered if I'd be doing my self any favors or just making things worse if I opted for that path. My goal would be to avoid even looking at JS at all whilst still being able to develop for the front end. Thoughts? Edit: no idea why you're being down voted. Perhaps you hurt the feelings of some JS developers? ;-)
fbn79 8y ago
I say unto you: one must still have chaos in oneself to be able to give birth to a dancing star. I say unto you: you still have chaos in yourselves. FN
rdiddly 8y ago
Literal LOL (LLOL? It'll catch on yet) at the existence of both is-odd and is-even!
suprfnk 8y ago
I am absolutely flabbergasted that the `is-odd` package on NPM [1] was downloaded 3 million times last week. In any project, a check for an odd number would half the time not even have its own _function_. Even an aggressive build server that redownloads and builds 50 times a day would require 60.000 separate installs... Does anyone have any reasonable explanation for this? [1] https://www.npmjs.com/package/is-odd https://www.npmjs.com/package/is-odd
- rypskar 8y ago
  The best part is that is-odd depends on is-even [1] [1]https://www.npmjs.com/package/is-even https://www.npmjs.com/package/is-even
  - flukus 8y ago
    Other way around, is-even depends on is-odd. But wait, there's more. Is even depends on is-number, so in the node.js world '515' is odd. It doesn't matter how good javascript compilers get when you throw this level of slop in.
    - pluma 8y ago
      As a JS dev, this actually offends me. I wouldn't expect a specialised package to silently convert strings to numbers. I think is-odd and friends are the aftermath of the microlibrary craze that followed the popularity of "toolbelt" libraries like jquery, underscore and lodash (which featured some equally inane methods but bundled them along with a load of far more complex helpers you wouldn't want to implement and maintain yourself).
  - grajaganDev 8y ago
    I found this funny but on further reflection it is just good engineering.
- LandR 8y ago
  This is should be funny, however in the JS / Web dev world nothing suprises me anyone. The whole ecosystem is an insane mess.
latchkey 8y ago
npm might even install a dependency without you realizing it (typo). https://github.com/npm/npm/issues/20072 https://github.com/npm/npm/issues/20072
emilsedgh 8y ago
I think Node.js ecosystem is like a big desert. There are true gems and treasures you can find but it's mostly useless.
politelemon 8y ago
Shouldn't the main concern highlighted here already be taken care of with package-lock.json and yarn.lock? They record the dependency trees, so a rogue package update should not really affect you.
- scintill76 8y ago
  Assuming you audit what was locked, and audit whenever you update, then yes. I'd be worried a coworker would go on an update spree just because they can, and wouldn't audit the new code.
  - ralusek 8y ago
    Yes but then you're no longer grappling with the limitations of the ecosystem. Introducing a rogue developer who updates dependencies irresponsibly is a danger in any language.
    - caspervonb 8y ago
      Who really audits every single dependency in their dependency tree? not so much a rogue developer, but most.
      - xfer 8y ago
        If you are hobby programming, then not really, but in most workplaces, dependencies are usually pinned and not updated until necessary(bug fixes mostly).
        caspervonb 8y ago
        From what I've seen as a contractor usually there is no package-lock at all.
kuschku 8y ago
What I'm interested in — most open source licenses such as MIT require that a copyright notice is preserved for each dependency, direct or transitive. Many licenses even require that such copyright notices for all dependencies are shown in an about menu in the software. How do you do that with node.js projects? Do you just have an about screen with 15'000 licenses and copyright attributions in your software? Do you just violate the license?
- caspervonb 8y ago
  For Node.js projects, the MIT clause should be covered by simply having the entire package installed, provided the author actually put the license file in the package. However when using a bundler; you are potentially violating the licenses of a bunch of packages.
  - kuschku 8y ago
    It also becomes an issue once you’re building a web application, as you’ll have to present it in the UI.
pluma 8y ago
Comparing is-odd with left-pad is a bit disingenuous. Left-pad had a fairly trivial implementation but that implementation is easy to get wrong, so it's something you want to make sure you have covered by tests and you want to define the edge cases. It's something that should be (and as of ES2017 is) part of the language and not something you want to keep reinventing (alongside all the edge cases and potential for annoying bugs). The is-odd and is-even packages OTOH are blatantly ridiculous. I had a look in my dependency tree and it seems is-odd made its way into it via a package called "nanomatch" which used is-odd because the maintainer apparently was of the opinion that (to paraphrase) "if it's difficult enough to get wrong on the first attempt it's difficult enough to warrant delegating to a dependency" -- which he seems to have taken to the logical extreme. That said, the package in question removed the dependency on is-odd after what seems to have been a lot of negativity about his decision not to inline the logic previously: https://github.com/micromatch/nanomatch/pull/11 https://github.com/micromatch/nanomatch/pull/11 so once that change propagates through the dependency tree, it should at least no longer show up for some people.
- twexler 8y ago
  > Left-pad had a fairly trivial implementation but that implementation is easy to get wrong, so it's something you want to make sure you have covered by tests ...What? That makes no sense. `left-pad` is trivial to implement and test. There may be edge cases but for most people, writing tests to cover the edge cases they care about rather than pulling in a dependency just to handle something as simple as padding a string. Not to mention, Node's near-complete lack of a standard library is at fault here, not developers, nor the ECMA technical committee.
  - pluma 8y ago
    > `left-pad` is trivial to implement and test There's a tangible difference between `value % 2 === 1` and this: https://github.com/stevemao/left-pad/blob/master/index.js https://github.com/stevemao/left-pad/blob/master/index.js You're not even contradicting me. I said it has a fairly trivial implementation and you concede that it has edge cases. You're just disagreeing on whether that warrants using an external dependency rather than writing it yourself (and writing all those tests yourself too). > Node's near-complete lack of a standard library is at fault here, not developers, nor the ECMA technical committee. So you think String.prototype.padStart should have been a Node built-in module rather than a language feature? Are we talking about the same language that also has String.prototype.italics and String.prototype.bold? There's nothing Node could or should have done about the lack of string padding. Node's "standard library" is first and foremost concerned with enabling network and filesystem IO. JS on the other hand even lacks a built-in way to handle dates properly (the Date class is largely an afterthought based on Java). String.prototype.padStart deserves to be a language feature and it should have been a language feature from the start. Most other languages provide this out of the box. The reason these things end up on NPM is not that JS developers are idiots but that they don't want to reimplement the same basic utility functions every other language would offer as a standard library or copy folders of code around. Yes, this is absurd when it comes to things the language actually DOES offer (or basic maths like `value % 2 === 1`) but for the most part NPM is fulfilling JS developers' need for a standard library -- and that's entirely on not just Node but also the language itself (and by extension TC39). EDIT: It's worth pointing out that left-pad has zero dependencies, as it should. Compare this to is-even (i.e. `value % 2 === 0` as a function) which depends on is-odd which depends on is-number -- all written by the same author and 100% serious.
jariz 8y ago
Yes, so let's all keep reinventing the wheel, over and over again. I know the author uses pretty extreme examples, but I still don't see what's inherently wrong with the mentioned packages. - it does what you think it does: `isOdd(2)` is a lot easier to understand than `value % 2` - even though most of these are one liners, it's one line of code less that you have to maintain - if the package is at least somewhat popular, it's highly likely it has measures for edge cases; stuff I wouldn't have thought of when writing it myself. On top of that, mentioning left-pad is really cliche and weak at this point (bear in mind this happend 2 years ago), npm has taken measures and nothing similar has happend since then, and even then, it was still a fairly isolated event - the whole thing actually took 10 minutes. I'd much rather take a rare, yet-to-happen-again chance that a package goes down, over having to rewrite simple utility functions over and over again, every month.
- caspervonb 8y ago
  > Yes, so let's all keep reinventing the wheel, over and over again. Writing basic code, like operator calls is not reinventing the wheel. > `isOdd(2)` is a lot easier to understand than `value % 2` It's not tho, an operator is fundamental and everyone knows what it does. The function on the other hand you have no clue and have to check the source to see what it does. > On top of that, mentioning left-pad is really cliche and weak at this point (bear in mind this happend 2 years ago), npm has taken measures and nothing similar has happend since then, and even then, it was still a fairly isolated event - the whole thing actually took 10 minutes. Mentioning left-pad because the culture has not changed since then, this is a culture problem and not a technical one IMHO.
- goatlover 8y ago
  > isOdd(2)` is a lot easier to understand than `value % 2` Isn't modulo something basic every programmer should know about?
  - cannedslime 8y ago
    is-odd package doesn't use modulus. And I don't think thats really the issue either. is-odd has three dependencies of its own, one being is-number...
gyaru 8y ago
>because no one locks down a dependency like is-string to a patch version. Is this actually true? Is this not what "package-lock.json" is intended to solve? (Not primarily a javascript developer)
jannes 8y ago
What disgusts me the most about the JS ecosystem these days is that even the most basic developer tools have tons of depencies. Let's say you want to use webpack and TypeScript on a project: Boom. 425 packages installed. And that's before you have written a single line of code. (It's probably worth noting that the offender is webpack. TypeScript seems to be dependency-free.) [1] Every single one of those 425 packages could have a malicious postinstall script that deletes your entire home directory or sends your private ssh keys to a remote server. npm makes it too easy to add dependencies to a library. [1]: If you also want webpack-cli that's another cool 441 packages for you.
- fmsf 8y ago
  It's interesting to compare with JVM ecosystem, where each dependency can only have one version, and you get version collisions if multiple packages list different versions of the same sub-dependency. I feel there is more thought in adding sub-dependencies, while in the node ecosystem because you won't have collisions, devs don't really seem to care how much they add.
- aolagers 8y ago
  This. Even with minimal applications your node_modules is so full of stuff that you need external tools for reviewing your dependency graph. But TypeScript is a positive exception to this trend. It has no (npm) dependencies at all.
  - kakwa_ 8y ago
    I remember trying to play a little with node and vue.js a few months ago (I'm a devops/backend developper by trade). Having seen the size of my node_modules directory, nearly 1GB, I was kind of horrified (to be fair, I checked most of the testing options, so a lot of the libraries were coming from that).
- frou_dh 8y ago
  "Small package" culture is Code Reuse writ large. Maybe Code Reuse is not the paradise the software industry has for so long thought it was?
z3t4 8y ago
This is utopia, with software virgins, all collaborating, sharing and reusing. Please don't destroy this paradise.
kabes 8y ago
So if I'm releasing an is-odd pip package, does that mean the python ecosystem sucks?
- strken 8y ago
  No, but if flask or pip or some other widely-used piece of the python ecosystem pulled in unmaintained packages, trivial packages, or very large package trees, it might make the python ecosystem chaotic and insecure.
- corrigible 8y ago
  Only if it gets 500k downloads per day :)
- deleted 8y ago
  [deleted]
naranha 8y ago
Almost all programming languages now have similar standard package managers. I still remember back in the day when everybody would joke about how Maven downloads the whole internet on the first build. This is still true today for Maven, NPM, pip, composer, you name it. You are downloading hundreds of packages for which you implicitly trust their maintainer to not do anything shady. In case of Maven - and likely most others - packages are not even digitally signed by the publisher and on top of it are binaries, that you cannot easily inspect. I'd argue NPM is slightly better than that, at least it (often) includes the source code, though you still have illegible instructions in form of minified/uglified code. I think NodeJS already has a decent stdlib and there will always be the need to pull dependencies in for basic functionality (apache commons, guava etc. in the Java world). That's just how OSS works. There should be a simple way to display and verify the chain of trust though.
- strken 8y ago
  NodeJS has a feature-rich standard library, but it's let down by poor ergonomics. Compare, for example, the built-in http library[0] with request-promise[1]. I think the difficulty of using the standard library makes NodeJS devs feel better about downloading lots of little helper libraries. [0] https://nodejs.org/api/http.html#http_http_request_options_callback https://nodejs.org/api/http.html#http_http_request_options_c... [1] https://github.com/request/request-promise#cheat-sheet https://github.com/request/request-promise#cheat-sheet
  - WorldMaker 8y ago
    I think a lot of it is NodeJS settling on its standard library a bit ahead of the larger language ecosystem itself. Promises were standardized shortly after Node bumbled into a callback-heavy design. The efforts on the ES2015 Module syntax occurred after Node settled for "CommonJS" which wasn't all that common at the time. Node's HTTP library as a specific example, particularly suffers from callbacks instead of promises. It also suffers here in that it also predates the Fetch standard making its way into browsers now. Hopefully the Node standard library will continue to converge with the overall JS ecosystem and catch up to the work that the Browsers have been leading here.
dyjakan 8y ago
Much more entertaining and same argument: https://hackernoon.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5 https://hackernoon.com/im-harvesting-credit-card-numbers-and...
Sir_Cmpwn 8y ago
This is a large part of why, in my opinion, language-specific package managers are generally bunk (though language-specific packaging tooling is not necessarily so). I install all of my Python packages system-wide through my distribution's package manager, and if no package is available I make one. Distribution packages are cryptographically signed and reviewed by a dispassionate human being before being published, and are available on mirrors around the world to mitigate the single source of failure language repositories tend to suffer from. Of course, this is difficult for something like Node, whose package ecosystem is out of control.
- badosu 8y ago
  Assuming we could avoid the legacy leverage, would it be possible to build or use an already existing package manager to perform this, wrapping all specific logic for each language?
  - Sir_Cmpwn 8y ago
    The human element is important imo. I don't think automating it away is wise. However, many distros have tools for speeding the work for particular languages.
Blackstone4 8y ago
What is the alternative? Have the large projects maintain their own dependencies? i.e. React, Webpack, Typescript, Express... Create and maintain your own packages? Review each package and save a copy in your own repo for re-use? Upgrading would be a pain.
- krapp 8y ago
  >Have the large projects maintain their own dependencies? i.e. React, Webpack, Typescript, Express... >Create and maintain your own packages? >Review each package and save a copy in your own repo for re-use? Yes? It is possible for things to be too convenient, IMHO. >Upgrading would be a pain. Upgrading should be at least a bit painful. Upgrading without pain means you don't really know what changes you're making to your codebase, and you don't care to know, you just want the magic black box to work and not bother you with the details. There should be at least more effort put into upgrades than the least amount possible.
kumarvvr 8y ago
Why would WebPack require to use a package like "is-odd" or "is-even"??
- caspervonb 8y ago
  Indirectly through it's vast dependency tree.

118 points by unicornode 8y ago

In my view the core problem is the complete lack of standards and interfaces around what constitutes a “package”, “module”, “library” or even “framework”. (It’s an industry-wide problem, not just JavaScript.) In npm’s single global namespace, “is-even” is a package. So is the Firebase SDK, even though the former is a single line of code and the latter is around 70 MB. But a package can also be something that’s not even JavaScript code: maybe a CSS “framework”, or that infamous dictionary text that gets replicated a dozen times in any large npm-dependent project because some popular package uses the text in its tests. If hardware parts were distributed like this, there would be no standard fittings, nothing would be categorized. A hardware store would just be an enormous pile of jumbled parts and tools. “Oh, you need a bolt? No problem, John will jump into the pile with scuba gear and bring you some packages that look suitable. It’s going to take a few days, hopefully we’ll find something that fits.”

treve 8y ago
What are you looking for exactly? A category system? The argument seems kind of weird. A bit similar to how a .zip file might contain just a few things, but also a lot of things. I think it's kind of a good thing that the same delivery mechanism can be used for various things, but I'm guessing you have a problem with discoverability?
- pavlov 8y ago
  The problem affects not just discoverability of new dependencies, but also development and maintenance of settled dependencies. When you see a line like this in package.json... "dependencies": { "foobarfrobblizer": "^0.9.31", There's no simple way to determine what foobarfrobblizer does, where it's actually used in the app, what side effects it may perform when called, what dependencies it pulls, why it was included, and whether it could be replaced with something else that offers a similar API. For a single package, I can figure all this out manually. But projects these days are seriously dependency-binged. If I use "create-react-app" to make a Hello World web app, it gives me 150 MB of stuff in node_modules. I'm never going to look at even 1% of those packages to see what's actually in there. I got started in programming when a dependency meant either a DLL file provided by someone else, or a static library that you manually link into your executable. Taking on a new dependency was a big decision: do I really want to rely on this DLL file being there for me, or having to carry this separately built binary blob in my project? I still get that same dread when I have to add an npm package to a project, but it doesn't seem like most people feel that way. On top of those DLL files 25 years ago, there were systems like Microsoft COM that let you use them interchangeably through standard interfaces. You could make a plugin UI component that works seamlessly in Visual Basic by simply conforming to a COM interface. It even worked from multiple languages: that UI component could be written in C, C++ or Object Pascal if you wanted. The web development ecosystem has a long way to go before it matches the usability of 1995 desktop development — and that's a painfully low bar.
  - askmike 8y ago
    > Taking on a new dependency was a big decision: do I really want to rely on this DLL file being there for me, or having to carry this separately built binary blob in my project? I'd argue that the NPM ecosystem (although not perfect, hence OP) is a lot more evolved specifically because of how easy it is to include a wide range of dependencies. The speed with which you can write software is an awful lot higher, and (if done properly) you can rely on safe dependencies that are tested and used by everyone else. Why reinvent the wheel in every project?
    - raarts 8y ago
      > you can rely on This is exactly the problem. Given the current ecosystem, you can't and shouldn't. You are just proving GP's point.
  - krapp 8y ago
    >There's no simple way to determine what foobarfrobblizer does, where it's actually used in the app, what side effects it may perform when called, what dependencies it pulls, why it was included, and whether it could be replaced with something else that offers a similar API. Isn't this an application issue? The manifest for each project should tell you that, it should be possible to build the dependency tree as a preview and see all of the metadata in aggregate.

sbr464 8y ago

Click bait, move along

movedx 8y ago
Are you are JavaScript developer, out of interest?
- sbr464 8y ago
  yes
  - movedx 8y ago
    I'm not the author of the article, but could you expand on why it's click bait? Personally, I think NPM is a crazy, mental concept. As a Go developer (so mostly server-side work) I can't fathom the amount of deps that are pulled from NPM when I'm working with or watching a JS code base being used. I assume server-side NodeJS code bases are the same their front end cousins in terms of deps pulled(?)
    - sbr464 8y ago
      Hi, appreciate the followup, apologies for the previous short comment. I feel like it's clickbait, not worth reading, for these points: 1. Even if the argument is valid, the points made in this article have been expressed, shared, (whined about?) probably 9+ times in my own reading. It's really time to move on. 2. There is nothing inherently wrong with using small/simple modules. Once you start to reuse code in many projects, the value becomes clear. Some are juicy and complex, some are eye-bleeding simple. It's better than copying and pasting code around folders. Use org modules/private modules and the problem is solved. I absolutely believe that npm is ripe for *potential abuse, although I have yet to have any issues. No one has time for Breitbart-esque fearmongering, or the regurgitation of stale points while an ample selection of real world problems exist to be solved. Jeez that reads harsh. Apologies, it's late.
      - deleted 8y ago

wyqydsyq 8y ago

yet-another-article-talking-about-the-left-pad-incident-and-fragile-dependency-chains-in-general-with-a-sensationalized-clickbait-title

movedx 8y ago
Are you are JavaScript developer, out of interest?
deleted 8y ago
[deleted]

baybal2 8y ago

The "modern" JS ecosystem: 1. Over-reliance on tooling. JS was made as a scripting language, if you can't use it as such, its purpose is lost. And that tooling works against betterment of the ecosystem - lib authors rely on tooling to fix issues of the platform, rather than seeking a definite solution. 2. Any serious JS project is a long term maintenance burden. The prevalence of "single line libraries" makes future-proofing of the code impossible. You don't only have to version-freeze your immediate dependencies, but dependencies of your dependencies. You will have to manage their version manually, it all becomes your own problem. 3. Authors of JS VMs swear on their firstborns that JS performance has been greatly improved, yet, it sucks. For practical purposes, JS code is barely as quick as perl or python. 4. Too much evals in 3rd party code everywhere. The only way you can secure your front-end is to ensure no third party code policy on your page. When it comes to the back-end, things become more grave. On the back-end, you can say that there is no universal solution. 5. Zoo of standards on the front-end, browser APIs changing on a whim. 6. Zoo of sublanguages: Coffeescript, LiveScript ClojureScript, MoonScript, AtScript, TypeScript. Having to mix anymuch of them in a single project is, you can guess, a supernightmare. Yet, I don't completely hate people who want to move JS forward. TC39 may well be too heavy on being staffed by theoreticians, but they have right priorities: standardization, having whatever going into the standard being highly dry before even approaching the pre-standard status, enforcing a semblance of synchronicity to the ecosystem. What JS needs in my opinion: 1. HARD VERSIONING 2. Compliance certification through testing with a hard version 3. An even harder enforced decoupling of the paper standard and implementation. While the majority of modern JS implementations are VM based, this should not make the standard to accommodate design decisions dictated by a VM implementation. 4. Readiness to do breaking changes in between major hard versions, and no reservations for backward compatibility. 5. Many browser APIs are de-facto parts of the standard library as such. They need to be managed as a part of the standard.

movedx 8y ago
Genuine question: does transpiling from Go, for example, make things any better? I've often wondered if I'd be doing my self any favors or just making things worse if I opted for that path. My goal would be to avoid even looking at JS at all whilst still being able to develop for the front end. Thoughts? Edit: no idea why you're being down voted. Perhaps you hurt the feelings of some JS developers? ;-)

fbn79 8y ago

I say unto you: one must still have chaos in oneself to be able to give birth to a dancing star. I say unto you: you still have chaos in yourselves. FN

rdiddly 8y ago

Literal LOL (LLOL? It'll catch on yet) at the existence of both is-odd and is-even!

suprfnk 8y ago

I am absolutely flabbergasted that the `is-odd` package on NPM [1] was downloaded 3 million times last week. In any project, a check for an odd number would half the time not even have its own _function_. Even an aggressive build server that redownloads and builds 50 times a day would require 60.000 separate installs... Does anyone have any reasonable explanation for this? [1] https://www.npmjs.com/package/is-odd https://www.npmjs.com/package/is-odd

rypskar 8y ago
The best part is that is-odd depends on is-even [1] [1]https://www.npmjs.com/package/is-even https://www.npmjs.com/package/is-even
- flukus 8y ago
  Other way around, is-even depends on is-odd. But wait, there's more. Is even depends on is-number, so in the node.js world '515' is odd. It doesn't matter how good javascript compilers get when you throw this level of slop in.
  - pluma 8y ago
    As a JS dev, this actually offends me. I wouldn't expect a specialised package to silently convert strings to numbers. I think is-odd and friends are the aftermath of the microlibrary craze that followed the popularity of "toolbelt" libraries like jquery, underscore and lodash (which featured some equally inane methods but bundled them along with a load of far more complex helpers you wouldn't want to implement and maintain yourself).
- grajaganDev 8y ago
  I found this funny but on further reflection it is just good engineering.
LandR 8y ago
This is should be funny, however in the JS / Web dev world nothing suprises me anyone. The whole ecosystem is an insane mess.

latchkey 8y ago

npm might even install a dependency without you realizing it (typo). https://github.com/npm/npm/issues/20072 https://github.com/npm/npm/issues/20072

emilsedgh 8y ago

I think Node.js ecosystem is like a big desert. There are true gems and treasures you can find but it's mostly useless.

politelemon 8y ago

Shouldn't the main concern highlighted here already be taken care of with package-lock.json and yarn.lock? They record the dependency trees, so a rogue package update should not really affect you.

scintill76 8y ago
Assuming you audit what was locked, and audit whenever you update, then yes. I'd be worried a coworker would go on an update spree just because they can, and wouldn't audit the new code.
- ralusek 8y ago
  Yes but then you're no longer grappling with the limitations of the ecosystem. Introducing a rogue developer who updates dependencies irresponsibly is a danger in any language.
  - caspervonb 8y ago
    Who really audits every single dependency in their dependency tree? not so much a rogue developer, but most.
    - xfer 8y ago
      If you are hobby programming, then not really, but in most workplaces, dependencies are usually pinned and not updated until necessary(bug fixes mostly).
      - caspervonb 8y ago
        From what I've seen as a contractor usually there is no package-lock at all.

kuschku 8y ago

What I'm interested in — most open source licenses such as MIT require that a copyright notice is preserved for each dependency, direct or transitive. Many licenses even require that such copyright notices for all dependencies are shown in an about menu in the software. How do you do that with node.js projects? Do you just have an about screen with 15'000 licenses and copyright attributions in your software? Do you just violate the license?

caspervonb 8y ago
For Node.js projects, the MIT clause should be covered by simply having the entire package installed, provided the author actually put the license file in the package. However when using a bundler; you are potentially violating the licenses of a bunch of packages.
- kuschku 8y ago
  It also becomes an issue once you’re building a web application, as you’ll have to present it in the UI.

pluma 8y ago

Comparing is-odd with left-pad is a bit disingenuous. Left-pad had a fairly trivial implementation but that implementation is easy to get wrong, so it's something you want to make sure you have covered by tests and you want to define the edge cases. It's something that should be (and as of ES2017 is) part of the language and not something you want to keep reinventing (alongside all the edge cases and potential for annoying bugs). The is-odd and is-even packages OTOH are blatantly ridiculous. I had a look in my dependency tree and it seems is-odd made its way into it via a package called "nanomatch" which used is-odd because the maintainer apparently was of the opinion that (to paraphrase) "if it's difficult enough to get wrong on the first attempt it's difficult enough to warrant delegating to a dependency" -- which he seems to have taken to the logical extreme. That said, the package in question removed the dependency on is-odd after what seems to have been a lot of negativity about his decision not to inline the logic previously: https://github.com/micromatch/nanomatch/pull/11 https://github.com/micromatch/nanomatch/pull/11 so once that change propagates through the dependency tree, it should at least no longer show up for some people.

twexler 8y ago
> Left-pad had a fairly trivial implementation but that implementation is easy to get wrong, so it's something you want to make sure you have covered by tests ...What? That makes no sense. `left-pad` is trivial to implement and test. There may be edge cases but for most people, writing tests to cover the edge cases they care about rather than pulling in a dependency just to handle something as simple as padding a string. Not to mention, Node's near-complete lack of a standard library is at fault here, not developers, nor the ECMA technical committee.
- pluma 8y ago
  > `left-pad` is trivial to implement and test There's a tangible difference between `value % 2 === 1` and this: https://github.com/stevemao/left-pad/blob/master/index.js https://github.com/stevemao/left-pad/blob/master/index.js You're not even contradicting me. I said it has a fairly trivial implementation and you concede that it has edge cases. You're just disagreeing on whether that warrants using an external dependency rather than writing it yourself (and writing all those tests yourself too). > Node's near-complete lack of a standard library is at fault here, not developers, nor the ECMA technical committee. So you think String.prototype.padStart should have been a Node built-in module rather than a language feature? Are we talking about the same language that also has String.prototype.italics and String.prototype.bold? There's nothing Node could or should have done about the lack of string padding. Node's "standard library" is first and foremost concerned with enabling network and filesystem IO. JS on the other hand even lacks a built-in way to handle dates properly (the Date class is largely an afterthought based on Java). String.prototype.padStart deserves to be a language feature and it should have been a language feature from the start. Most other languages provide this out of the box. The reason these things end up on NPM is not that JS developers are idiots but that they don't want to reimplement the same basic utility functions every other language would offer as a standard library or copy folders of code around. Yes, this is absurd when it comes to things the language actually DOES offer (or basic maths like `value % 2 === 1`) but for the most part NPM is fulfilling JS developers' need for a standard library -- and that's entirely on not just Node but also the language itself (and by extension TC39). EDIT: It's worth pointing out that left-pad has zero dependencies, as it should. Compare this to is-even (i.e. `value % 2 === 0` as a function) which depends on is-odd which depends on is-number -- all written by the same author and 100% serious.

jariz 8y ago

Yes, so let's all keep reinventing the wheel, over and over again. I know the author uses pretty extreme examples, but I still don't see what's inherently wrong with the mentioned packages. - it does what you think it does: `isOdd(2)` is a lot easier to understand than `value % 2` - even though most of these are one liners, it's one line of code less that you have to maintain - if the package is at least somewhat popular, it's highly likely it has measures for edge cases; stuff I wouldn't have thought of when writing it myself. On top of that, mentioning left-pad is really cliche and weak at this point (bear in mind this happend 2 years ago), npm has taken measures and nothing similar has happend since then, and even then, it was still a fairly isolated event - the whole thing actually took 10 minutes. I'd much rather take a rare, yet-to-happen-again chance that a package goes down, over having to rewrite simple utility functions over and over again, every month.

caspervonb 8y ago
> Yes, so let's all keep reinventing the wheel, over and over again. Writing basic code, like operator calls is not reinventing the wheel. > `isOdd(2)` is a lot easier to understand than `value % 2` It's not tho, an operator is fundamental and everyone knows what it does. The function on the other hand you have no clue and have to check the source to see what it does. > On top of that, mentioning left-pad is really cliche and weak at this point (bear in mind this happend 2 years ago), npm has taken measures and nothing similar has happend since then, and even then, it was still a fairly isolated event - the whole thing actually took 10 minutes. Mentioning left-pad because the culture has not changed since then, this is a culture problem and not a technical one IMHO.
goatlover 8y ago
> isOdd(2)` is a lot easier to understand than `value % 2` Isn't modulo something basic every programmer should know about?
- cannedslime 8y ago
  is-odd package doesn't use modulus. And I don't think thats really the issue either. is-odd has three dependencies of its own, one being is-number...

gyaru 8y ago

>because no one locks down a dependency like is-string to a patch version. Is this actually true? Is this not what "package-lock.json" is intended to solve? (Not primarily a javascript developer)

jannes 8y ago

What disgusts me the most about the JS ecosystem these days is that even the most basic developer tools have tons of depencies. Let's say you want to use webpack and TypeScript on a project: Boom. 425 packages installed. And that's before you have written a single line of code. (It's probably worth noting that the offender is webpack. TypeScript seems to be dependency-free.) [1] Every single one of those 425 packages could have a malicious postinstall script that deletes your entire home directory or sends your private ssh keys to a remote server. npm makes it too easy to add dependencies to a library. [1]: If you also want webpack-cli that's another cool 441 packages for you.

fmsf 8y ago
It's interesting to compare with JVM ecosystem, where each dependency can only have one version, and you get version collisions if multiple packages list different versions of the same sub-dependency. I feel there is more thought in adding sub-dependencies, while in the node ecosystem because you won't have collisions, devs don't really seem to care how much they add.
aolagers 8y ago
This. Even with minimal applications your node_modules is so full of stuff that you need external tools for reviewing your dependency graph. But TypeScript is a positive exception to this trend. It has no (npm) dependencies at all.
- kakwa_ 8y ago
  I remember trying to play a little with node and vue.js a few months ago (I'm a devops/backend developper by trade). Having seen the size of my node_modules directory, nearly 1GB, I was kind of horrified (to be fair, I checked most of the testing options, so a lot of the libraries were coming from that).
frou_dh 8y ago
"Small package" culture is Code Reuse writ large. Maybe Code Reuse is not the paradise the software industry has for so long thought it was?

z3t4 8y ago

This is utopia, with software virgins, all collaborating, sharing and reusing. Please don't destroy this paradise.

kabes 8y ago

So if I'm releasing an is-odd pip package, does that mean the python ecosystem sucks?

strken 8y ago
No, but if flask or pip or some other widely-used piece of the python ecosystem pulled in unmaintained packages, trivial packages, or very large package trees, it might make the python ecosystem chaotic and insecure.
corrigible 8y ago
Only if it gets 500k downloads per day :)
deleted 8y ago
[deleted]

naranha 8y ago

Almost all programming languages now have similar standard package managers. I still remember back in the day when everybody would joke about how Maven downloads the whole internet on the first build. This is still true today for Maven, NPM, pip, composer, you name it. You are downloading hundreds of packages for which you implicitly trust their maintainer to not do anything shady. In case of Maven - and likely most others - packages are not even digitally signed by the publisher and on top of it are binaries, that you cannot easily inspect. I'd argue NPM is slightly better than that, at least it (often) includes the source code, though you still have illegible instructions in form of minified/uglified code. I think NodeJS already has a decent stdlib and there will always be the need to pull dependencies in for basic functionality (apache commons, guava etc. in the Java world). That's just how OSS works. There should be a simple way to display and verify the chain of trust though.

strken 8y ago
NodeJS has a feature-rich standard library, but it's let down by poor ergonomics. Compare, for example, the built-in http library[0] with request-promise[1]. I think the difficulty of using the standard library makes NodeJS devs feel better about downloading lots of little helper libraries. [0] https://nodejs.org/api/http.html#http_http_request_options_callback https://nodejs.org/api/http.html#http_http_request_options_c... [1] https://github.com/request/request-promise#cheat-sheet https://github.com/request/request-promise#cheat-sheet
- WorldMaker 8y ago
  I think a lot of it is NodeJS settling on its standard library a bit ahead of the larger language ecosystem itself. Promises were standardized shortly after Node bumbled into a callback-heavy design. The efforts on the ES2015 Module syntax occurred after Node settled for "CommonJS" which wasn't all that common at the time. Node's HTTP library as a specific example, particularly suffers from callbacks instead of promises. It also suffers here in that it also predates the Fetch standard making its way into browsers now. Hopefully the Node standard library will continue to converge with the overall JS ecosystem and catch up to the work that the Browsers have been leading here.

dyjakan 8y ago

Much more entertaining and same argument: https://hackernoon.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5 https://hackernoon.com/im-harvesting-credit-card-numbers-and...

Sir_Cmpwn 8y ago

This is a large part of why, in my opinion, language-specific package managers are generally bunk (though language-specific packaging tooling is not necessarily so). I install all of my Python packages system-wide through my distribution's package manager, and if no package is available I make one. Distribution packages are cryptographically signed and reviewed by a dispassionate human being before being published, and are available on mirrors around the world to mitigate the single source of failure language repositories tend to suffer from. Of course, this is difficult for something like Node, whose package ecosystem is out of control.

badosu 8y ago
Assuming we could avoid the legacy leverage, would it be possible to build or use an already existing package manager to perform this, wrapping all specific logic for each language?
- Sir_Cmpwn 8y ago
  The human element is important imo. I don't think automating it away is wise. However, many distros have tools for speeding the work for particular languages.

Blackstone4 8y ago

What is the alternative? Have the large projects maintain their own dependencies? i.e. React, Webpack, Typescript, Express... Create and maintain your own packages? Review each package and save a copy in your own repo for re-use? Upgrading would be a pain.

krapp 8y ago
>Have the large projects maintain their own dependencies? i.e. React, Webpack, Typescript, Express... >Create and maintain your own packages? >Review each package and save a copy in your own repo for re-use? Yes? It is possible for things to be too convenient, IMHO. >Upgrading would be a pain. Upgrading should be at least a bit painful. Upgrading without pain means you don't really know what changes you're making to your codebase, and you don't care to know, you just want the magic black box to work and not bother you with the details. There should be at least more effort put into upgrades than the least amount possible.

kumarvvr 8y ago

Why would WebPack require to use a package like "is-odd" or "is-even"??

caspervonb 8y ago
Indirectly through it's vast dependency tree.