7 ms·

This paper lends significant legitimacy to a casual observation that I've been concerned about for a long time: as the standard for what deserves to be a module

This paper lends significant legitimacy to a casual observation that I've been concerned about for a long time: as the standard for what deserves to be a module gets ever-lowered, the law of diminishing returns kicks in really hard. The package managers for Ruby, C#, Perl, Python etc offer ~100k modules. This offers strong evidence that most developer ecosystems produce (and occasionally maintain) a predictable number of useful Things. If npm has 750k+ modules available, that suggests that the standard for what constitutes a valuable quantity of functionality is 7.5X lower in the JS community. Given that every dependency increases your potential for multi-dimensional technical risk, this seems like it should be cause for reflection. It's not an abstract risk, either... as anyone who used left-pad circa 2016 can attest. When I create a new Rails 5.2 app, the dependency tree is 70 gems and most of them are stable to mature. When I create-react-app and see that there's 1014 items in node_modules, I have no idea what most of them actually do. And let's not forget: that's just the View layer of your fancy JS app.

mschuster91 8y ago
> If npm has 750k+ modules available, that suggests that the standard for what constitutes a valuable quantity of functionality is 7.5X lower in the JS community. Or that JS lacks a decent stdlib! As evidenced by the left-pad fiasco, JS simply lacks things that are either available by default in other languages or easily include-able from the system libc (in c/c++).
- maxxxxx 8y ago
  Totally agree. If JS had only a decent standard lib for string, collections and math I bet the number of packages would be reduced by a lot. I wonder what the reception of such a thing in NPM would be. Would people appreciate having only one, but larger dependency?
  - peteforde 8y ago
    There's an old engineering joke about how there's 13 incompatible power socket standards in use on Earth today. Sensing an opportunity to engineer a better solution, they created a power socket that is simpler, easier to use and cheaper to install than all current standards. Now there are 14 power socket standards.
    - maxxxxx 8y ago
      I know that one but I don't think it applies here. In JS there really doesn't seem to be a comprehensive library. When I look at a lot of NPM packages they are simple 5-10 line functions. Consolidating them into a larger library would help a lo in my view.
jrochkind1 8y ago
When I create a new rails 5.2.2 app, I see 79 dependencies in entire tree. Which is about what you said, and a lot less than 1014, sure. There are various reasons other than "low standards" that the JS ecosystem has developed to encourage even more massive dependency trees. One is that JS projects like this are delivered to the browser. If I want one function from, say, underscore (remember that?), but depend on all of it... do i end up shipping all of it to the browser to use one function? That would be unfortunate. Newer tools mean not necessarily, but it can be tricky, and some of this culture developed before those tools. But from this can develop a community culture of why _shouldn't_ I minimize the weight of dependencies? If some people only want one function and others only another, shouldn't they be separate dependencies so they can do that? And not expose themselves to possible bugs or security problems in all that other code they don't want? If dependencies can be dangeorus... isn't it better to have _surgical_ dependencies including only exactly what you need so you can take less of them? (Makes sense at first, but of course when you have 1000 of those "surgical" dependencies, it kind of breaks down). Another, like someone else said, is that JS in the browser has very little stdlib/built in functions. Another is tooling. The dependency trees were getting unmanageable in ruby before bundler was created (which inspired advanced dependency management features in most of the rest subsequent). We probably couldn't have as many dependencies as even Rails has without bundler. Your dependency complexity is limited by tooling support; but then when tooling support comes, it gives you a whole new level of dependency management problems that come with the crazy things the tooling let you do. These things all feed back on each other back and forth. I'm not saying it isn't giving rise to very real problems. But it's not just an issue of people having low standards or something.
- peteforde 8y ago
  You are correct, sir: 79 dependencies. Need more coffee! The influence of Bundler is just another example of Yehuda Katz doing something initially perceived as unpopular having a massive long-term impact on developer ecosystems. He is the Erdős of the web dev world. I wish he would sell options on his future endeavours like David Bowie did. Anyhow, in trying to keep my initial comment relatively brief, I held back on several points; what I really and truly don't get about the 750k thing is how anyone can track the libraries available. You know... with their brains. In Rails, gems are big ideas: authentication, an ORM, the ability to address all of AWS, send tweets. The idea that someone would even think to publish a "left-pad" (which I understand is just missing from the also-missing stdlib) as something people should import is what seems crazy. What makes you stop and think... wait, I'm typing too many characters, I need to see if there's anything on Github that can insert spaces at the beginning of a string. How would you even know what it's called? Is there a module for concatenating two strings? When does it become silly? How is it possible that finding a library to add padding which may or may not exist, doing even cursory code review and integrating it not take longer than just writing a few lines of code? Using the example of the escape-string-regexp module mentioned in the whitepaper... this would be a deeply flawed thing to add to your project. It has a hard-coded error message that has zero affordances for localization strategies. It is, at best, a few random lines from someone's hobby app. If every one of those 1014 modules in an empty project have weird, unknown-unknown failure modes, that sounds like a recipe for trouble. Finally, 1014 packages getting frequently version bumped is way less reliable in terms of unfortunate conflicts. Your surface area of all things that can go wrong shoots up and to the right... all to save a few lines of code?
gr__or 8y ago
> that suggests that the standard for what constitutes a valuable quantity of functionality is 7.5X lower in the JS community. Err, does it? The first thing this suggests is that the JS community is 7.5x larger (true for Ruby, even larger factor for Perl and not true for Pyhthon [1]). The second thing this suggests to me is that npm is X times more usable than those languages package managers, which from my experience true for Python (not for Ruby though, dunno about Perl). 1 - https://insights.stackoverflow.com/survey/2018/#technology https://insights.stackoverflow.com/survey/2018/#technology
- peteforde 8y ago
  I'm not trying be argumentative, but I honestly don't see how you conclude that 7.5x more packages === 7.5x more usuable. I don't have a huge amount of experience with pip or artisan, but they seemed roughly similar to both rpm and bundler in terms of concept and operation. You have a file with a list of modules to require, an executable tool that downloads all of those modules and their dependencies and finally a generated file that tracks the snapshot state of all dependencies. I don't perceive npm bringing 7.5x game to that evaluation. Don't get me wrong: I'm thrilled that it's working for so many people. Java was officially working for people when Rails became popular, too. However, popularity is a dangerous way to measure whether something is actually objectively better. Sometimes the most popular thing really is better. And sometimes Putin is super popular. It's complicated. :)

mschuster91 8y ago

> If npm has 750k+ modules available, that suggests that the standard for what constitutes a valuable quantity of functionality is 7.5X lower in the JS community. Or that JS lacks a decent stdlib! As evidenced by the left-pad fiasco, JS simply lacks things that are either available by default in other languages or easily include-able from the system libc (in c/c++).

maxxxxx 8y ago
Totally agree. If JS had only a decent standard lib for string, collections and math I bet the number of packages would be reduced by a lot. I wonder what the reception of such a thing in NPM would be. Would people appreciate having only one, but larger dependency?
- peteforde 8y ago
  There's an old engineering joke about how there's 13 incompatible power socket standards in use on Earth today. Sensing an opportunity to engineer a better solution, they created a power socket that is simpler, easier to use and cheaper to install than all current standards. Now there are 14 power socket standards.
  - maxxxxx 8y ago
    I know that one but I don't think it applies here. In JS there really doesn't seem to be a comprehensive library. When I look at a lot of NPM packages they are simple 5-10 line functions. Consolidating them into a larger library would help a lo in my view.

jrochkind1 8y ago

When I create a new rails 5.2.2 app, I see 79 dependencies in entire tree. Which is about what you said, and a lot less than 1014, sure. There are various reasons other than "low standards" that the JS ecosystem has developed to encourage even more massive dependency trees. One is that JS projects like this are delivered to the browser. If I want one function from, say, underscore (remember that?), but depend on all of it... do i end up shipping all of it to the browser to use one function? That would be unfortunate. Newer tools mean not necessarily, but it can be tricky, and some of this culture developed before those tools. But from this can develop a community culture of why _shouldn't_ I minimize the weight of dependencies? If some people only want one function and others only another, shouldn't they be separate dependencies so they can do that? And not expose themselves to possible bugs or security problems in all that other code they don't want? If dependencies can be dangeorus... isn't it better to have _surgical_ dependencies including only exactly what you need so you can take less of them? (Makes sense at first, but of course when you have 1000 of those "surgical" dependencies, it kind of breaks down). Another, like someone else said, is that JS in the browser has very little stdlib/built in functions. Another is tooling. The dependency trees were getting unmanageable in ruby before bundler was created (which inspired advanced dependency management features in most of the rest subsequent). We probably couldn't have as many dependencies as even Rails has without bundler. Your dependency complexity is limited by tooling support; but then when tooling support comes, it gives you a whole new level of dependency management problems that come with the crazy things the tooling let you do. These things all feed back on each other back and forth. I'm not saying it isn't giving rise to very real problems. But it's not just an issue of people having low standards or something.

peteforde 8y ago
You are correct, sir: 79 dependencies. Need more coffee! The influence of Bundler is just another example of Yehuda Katz doing something initially perceived as unpopular having a massive long-term impact on developer ecosystems. He is the Erdős of the web dev world. I wish he would sell options on his future endeavours like David Bowie did. Anyhow, in trying to keep my initial comment relatively brief, I held back on several points; what I really and truly don't get about the 750k thing is how anyone can track the libraries available. You know... with their brains. In Rails, gems are big ideas: authentication, an ORM, the ability to address all of AWS, send tweets. The idea that someone would even think to publish a "left-pad" (which I understand is just missing from the also-missing stdlib) as something people should import is what seems crazy. What makes you stop and think... wait, I'm typing too many characters, I need to see if there's anything on Github that can insert spaces at the beginning of a string. How would you even know what it's called? Is there a module for concatenating two strings? When does it become silly? How is it possible that finding a library to add padding which may or may not exist, doing even cursory code review and integrating it not take longer than just writing a few lines of code? Using the example of the escape-string-regexp module mentioned in the whitepaper... this would be a deeply flawed thing to add to your project. It has a hard-coded error message that has zero affordances for localization strategies. It is, at best, a few random lines from someone's hobby app. If every one of those 1014 modules in an empty project have weird, unknown-unknown failure modes, that sounds like a recipe for trouble. Finally, 1014 packages getting frequently version bumped is way less reliable in terms of unfortunate conflicts. Your surface area of all things that can go wrong shoots up and to the right... all to save a few lines of code?

gr__or 8y ago

> that suggests that the standard for what constitutes a valuable quantity of functionality is 7.5X lower in the JS community. Err, does it? The first thing this suggests is that the JS community is 7.5x larger (true for Ruby, even larger factor for Perl and not true for Pyhthon [1]). The second thing this suggests to me is that npm is X times more usable than those languages package managers, which from my experience true for Python (not for Ruby though, dunno about Perl). 1 - https://insights.stackoverflow.com/survey/2018/#technology https://insights.stackoverflow.com/survey/2018/#technology

peteforde 8y ago
I'm not trying be argumentative, but I honestly don't see how you conclude that 7.5x more packages === 7.5x more usuable. I don't have a huge amount of experience with pip or artisan, but they seemed roughly similar to both rpm and bundler in terms of concept and operation. You have a file with a list of modules to require, an executable tool that downloads all of those modules and their dependencies and finally a generated file that tracks the snapshot state of all dependencies. I don't perceive npm bringing 7.5x game to that evaluation. Don't get me wrong: I'm thrilled that it's working for so many people. Java was officially working for people when Rails became popular, too. However, popularity is a dangerous way to measure whether something is actually objectively better. Sometimes the most popular thing really is better. And sometimes Putin is super popular. It's complicated. :)