3 ms·

What's up with node people packaging every function into a module? How can anyone think that a clean build needing to pull in a dozen thousand files and taking

What's up with node people packaging every function into a module? How can anyone think that a clean build needing to pull in a dozen thousand files and taking twenty minutes is a good thing? Why have a, say, sha1 module instead of a general crypto one? I've seen at least one module that wasn't more than ten lines of actual code. It's packaging JSON and "tests" were far bigger. It was something trivial that should be in a stdlib.

sehr 12y ago
I don't need the rest of the stuff in the crypto module
mattdesl 12y ago
There are a lot of reasons to break modules into really tiny pieces, even if they are only a dozen lines or less. Some benefits: - a terse and frozen API (like "domready" and "xtend") does not end up with the scope creep and bitrot that monolithic frameworks and "standard libraries" tend to carry - it encourages diversity rather than "this is the one true way" - it generally leads to less fragmentation overall (and tighter and more robust apps) - each piece can be versioned, tested and bug-tracked independently - once you get used to it and start finding modules you like, it can be incredibly easy to prototype and rapidly iterate with existing solutions. My 100+ modules are in a similar domain (graphics) and my efficiency for prototyping has improved because of them. - it is better for reusability. If you have an algorithm that depends on jQuery or another monolithic framework for just a single function, it is hard to reuse. (ie version issues, bundle size) It took me a while to come around, but npm has really given me a better appreciation for small modules. :)
- STRML 12y ago
  Just to add to your list, small modules are incredibly useful for browserify/webpack apps. For example, if I just want to md5 some strings and do a `require('crypto')`, I've just pulled about 100 KB of code into my app. Instead, I can find some simple md5 lib (in this case I used one called "js-md5"), and get the same functionality in about 3KB. It seems a lot of Node projects have gone the small module direction. Given that npm is a package manager that (mostly) works correctly - which is so much harder than it sounds - we can actually use small modules in our app to little detriment. Yeah, an npm install might take a little bit longer than you'd like, although it's really not that slow. The duplication of modules isn't really an issue in server-side apps, and in webpack apps you can dedupe code pretty easily with webpack.optimize.DedupePlugin + gzipping, so it's not an issue there either.
  - MichaelGG 12y ago
    That sounds like poor tooling in JS, something solved by say, linkers, in other systems. And I'm not exaggerating about build times. A simple grunt build doing some basic template stuff would take about 20 minutes. The majority of that time was bringing in the ~13,000 files a rather simple static website needed to build. I ended up tossing the idea of independent builds and just made a persistent build machine that symlinked in node_modules. I've got a million lines of C program that takes less time to fully compile and link.
    - mattdesl 12y ago
      I would be curious to see what kind of tool could do this. It sounds like "tree shaking" (like dead code removal) which may be possible in the far future with ES6 imports. I imagine it would further add to build times. Not sure how your rant about grunt tasks and templating relates to small npm modules. A 20 minute build time sounds like something was vey wrong. My browserify (incremental) build time is < 100 ms which I can handle.
      - MichaelGG 12y ago
        Yeah perhaps current JS tools can't do dead code elimination because of the highly dynamic environment? Each module could provide some metadata on what it actually needs. I only target JS via cross compilers so I've never dealt with this problem. The small module system ends up requiring a ton of files, which is slow. Incremental builds don't really apply to a clean build server where you are basically doing "git clone ... && make". The actual processing isn't my complaint, just the enormous overhead npm's style imposes. I mentioned grunt since just having that plus uglify or so ends up bringing in 13k files or something.
danneu 12y ago
On the other side of the spectrum, you have large Java libraries where you need to somehow figure out which 20 classes collaborate to do a task, and the "documentation" link just drops you into one of those iframe-sidebar javadoc browsers. https://www.bouncycastle.org/docs/docs1.5on/index.html https://www.bouncycastle.org/docs/docs1.5on/index.html I really don't think the Node ecosystem got this one wrong.
- MichaelGG 12y ago
  That's the lack of design of a library, not really necessarily the size. You could easily have one entry point class with a bunch of ready to use functions and start your docs off there. Otherwise, you're just punting the discovery problem up a level. There's also a benefit in having multiple things in a module. You could make your code generic over the type of hash being used for instance. As far as node, you could easily export 20 different hash functions for use, instead of a dedicated sha1 package. Then getting yet another when you want hmac. The downside is nontrivial. In a repeatable build environment it can add tens of minutes of delays to a build as these tens of thousands of files aren't free. (And that was on a VM by one of the big providers, with top notch bandwidth.) Just even doing a local copy of so many files can take a while.
  - insaneirish 12y ago
    > In a repeatable build environment it can add tens of minutes of delays to a build as these tens of thousands of files aren't free. (And that was on a VM by one of the big providers, with top notch bandwidth.) Why would you be pulling modules in from an Internet based repo on every build? > Just even doing a local copy of so many files can take a while. That means your cloud provider has poor file I/O performance, which is not unusual.

sehr 12y ago

I don't need the rest of the stuff in the crypto module

mattdesl 12y ago

There are a lot of reasons to break modules into really tiny pieces, even if they are only a dozen lines or less. Some benefits: - a terse and frozen API (like "domready" and "xtend") does not end up with the scope creep and bitrot that monolithic frameworks and "standard libraries" tend to carry - it encourages diversity rather than "this is the one true way" - it generally leads to less fragmentation overall (and tighter and more robust apps) - each piece can be versioned, tested and bug-tracked independently - once you get used to it and start finding modules you like, it can be incredibly easy to prototype and rapidly iterate with existing solutions. My 100+ modules are in a similar domain (graphics) and my efficiency for prototyping has improved because of them. - it is better for reusability. If you have an algorithm that depends on jQuery or another monolithic framework for just a single function, it is hard to reuse. (ie version issues, bundle size) It took me a while to come around, but npm has really given me a better appreciation for small modules. :)

STRML 12y ago
Just to add to your list, small modules are incredibly useful for browserify/webpack apps. For example, if I just want to md5 some strings and do a `require('crypto')`, I've just pulled about 100 KB of code into my app. Instead, I can find some simple md5 lib (in this case I used one called "js-md5"), and get the same functionality in about 3KB. It seems a lot of Node projects have gone the small module direction. Given that npm is a package manager that (mostly) works correctly - which is so much harder than it sounds - we can actually use small modules in our app to little detriment. Yeah, an npm install might take a little bit longer than you'd like, although it's really not that slow. The duplication of modules isn't really an issue in server-side apps, and in webpack apps you can dedupe code pretty easily with webpack.optimize.DedupePlugin + gzipping, so it's not an issue there either.
- MichaelGG 12y ago
  That sounds like poor tooling in JS, something solved by say, linkers, in other systems. And I'm not exaggerating about build times. A simple grunt build doing some basic template stuff would take about 20 minutes. The majority of that time was bringing in the ~13,000 files a rather simple static website needed to build. I ended up tossing the idea of independent builds and just made a persistent build machine that symlinked in node_modules. I've got a million lines of C program that takes less time to fully compile and link.
  - mattdesl 12y ago
    I would be curious to see what kind of tool could do this. It sounds like "tree shaking" (like dead code removal) which may be possible in the far future with ES6 imports. I imagine it would further add to build times. Not sure how your rant about grunt tasks and templating relates to small npm modules. A 20 minute build time sounds like something was vey wrong. My browserify (incremental) build time is < 100 ms which I can handle.
    - MichaelGG 12y ago
      Yeah perhaps current JS tools can't do dead code elimination because of the highly dynamic environment? Each module could provide some metadata on what it actually needs. I only target JS via cross compilers so I've never dealt with this problem. The small module system ends up requiring a ton of files, which is slow. Incremental builds don't really apply to a clean build server where you are basically doing "git clone ... && make". The actual processing isn't my complaint, just the enormous overhead npm's style imposes. I mentioned grunt since just having that plus uglify or so ends up bringing in 13k files or something.

danneu 12y ago

On the other side of the spectrum, you have large Java libraries where you need to somehow figure out which 20 classes collaborate to do a task, and the "documentation" link just drops you into one of those iframe-sidebar javadoc browsers. https://www.bouncycastle.org/docs/docs1.5on/index.html https://www.bouncycastle.org/docs/docs1.5on/index.html I really don't think the Node ecosystem got this one wrong.

MichaelGG 12y ago
That's the lack of design of a library, not really necessarily the size. You could easily have one entry point class with a bunch of ready to use functions and start your docs off there. Otherwise, you're just punting the discovery problem up a level. There's also a benefit in having multiple things in a module. You could make your code generic over the type of hash being used for instance. As far as node, you could easily export 20 different hash functions for use, instead of a dedicated sha1 package. Then getting yet another when you want hmac. The downside is nontrivial. In a repeatable build environment it can add tens of minutes of delays to a build as these tens of thousands of files aren't free. (And that was on a VM by one of the big providers, with top notch bandwidth.) Just even doing a local copy of so many files can take a while.
- insaneirish 12y ago
  > In a repeatable build environment it can add tens of minutes of delays to a build as these tens of thousands of files aren't free. (And that was on a VM by one of the big providers, with top notch bandwidth.) Why would you be pulling modules in from an Internet based repo on every build? > Just even doing a local copy of so many files can take a while. That means your cloud provider has poor file I/O performance, which is not unusual.