3 ms·

It's mostly Babel and other build tools: https://gist.github.com/tlrobinson/1fba66be86f473ffe76312c32513c301 https://gist.github.com/tlrobinson/1fba66be86f473ff

It's mostly Babel and other build tools: https://gist.github.com/tlrobinson/1fba66be86f473ffe76312c32513c301 https://gist.github.com/tlrobinson/1fba66be86f473ffe76312c32... With npm every dependency gets it's own tree of dependencies, so there's tons of duplication. Also, this is using npm 2. With npm 3 that's approximately cut in half since it tries to flatten the hierarchy as much as possible, but I haven't switched to npm 3 yet (seems slower and still somewhat buggy)

jerf 11y ago
"With npm every dependency gets it's own tree of dependencies, so there's tons of duplication." That sounds like the missing bit of info. If that is the primary problem, then the good news is that git actually ought to be able to re-de-dupe that stuff back down; checking it into git may not be as catastrophic as you'd initially think.
- tlrobinson 11y ago
  Good point. If I add my node_modules to a new git repo it ends up being about 65MB, which is better, but still significant, and I imagine the overhead of tracking an additional 100,000+ files is non-trivial.
- etblg 11y ago
  That also shouldn't be true in the latest npm, which uses a global node_modules and a bit of trickery for different versions of the same library, instead of recursive node_modules.
nostrademons 11y ago
Babel seems to be a terrible offender as far as dependencies go. In my node_modules, babel-preset-es2015 takes up 200M - as you've noticed, each individual transformer has its own copy of the core Babel runtime. And Babel was largely responsible for this whole left-pad fiasco in the first place: about 2/3 of left-pad downloads were through Babel. The good news is that Node 6.0 is coming out next month, with V8 5.0, which supports 93% of ES6 natively. That means that most of the transformers can go away for server & Chrome code, with the appropriate Babel preset. And the big remaining ones - modules and async/await - are scheduled for Q2 2016, with work on modules already underway as of a couple weeks ago. So within a couple months, we may not need Babel at all anymore.
- nilliams 11y ago
  Yes, if you only write JS for Chrome and Node, not sure how many JS devs fall into that bracket!
  - nostrademons 11y ago
    You also have the option of doing split JS bundles, using different Babel presets, for different browsers. Eg. for Chrome the only thing you need is module imports, but for Safari you use a preset that also includes destructuring, function defaults, etc. This reduces the attack surface that a rogue Babel dependency has access to - instead of being able to pwn your server and all browsers, it can only pwn the browsers that use the particular transform that depends on it. It also can result in significant byte-size savings in the downloaded bundle.
    - deleted 11y ago
      [deleted]
- insin 11y ago
  It's specifically a Babel 6 problem - the way plugins are published means it now implicitly requires npm3. Similarly, I had to bail on installing the 'victory' package (React visualisation components) using npm2 more than 600MB into it attempting to pull down its dependencies - I don't know if this is still the case, but every "victory-*" plugin was installing its own copies of Babel 6 and Webpack...

jerf 11y ago

"With npm every dependency gets it's own tree of dependencies, so there's tons of duplication." That sounds like the missing bit of info. If that is the primary problem, then the good news is that git actually ought to be able to re-de-dupe that stuff back down; checking it into git may not be as catastrophic as you'd initially think.

tlrobinson 11y ago
Good point. If I add my node_modules to a new git repo it ends up being about 65MB, which is better, but still significant, and I imagine the overhead of tracking an additional 100,000+ files is non-trivial.
etblg 11y ago
That also shouldn't be true in the latest npm, which uses a global node_modules and a bit of trickery for different versions of the same library, instead of recursive node_modules.

nostrademons 11y ago

Babel seems to be a terrible offender as far as dependencies go. In my node_modules, babel-preset-es2015 takes up 200M - as you've noticed, each individual transformer has its own copy of the core Babel runtime. And Babel was largely responsible for this whole left-pad fiasco in the first place: about 2/3 of left-pad downloads were through Babel. The good news is that Node 6.0 is coming out next month, with V8 5.0, which supports 93% of ES6 natively. That means that most of the transformers can go away for server & Chrome code, with the appropriate Babel preset. And the big remaining ones - modules and async/await - are scheduled for Q2 2016, with work on modules already underway as of a couple weeks ago. So within a couple months, we may not need Babel at all anymore.

nilliams 11y ago
Yes, if you only write JS for Chrome and Node, not sure how many JS devs fall into that bracket!
- nostrademons 11y ago
  You also have the option of doing split JS bundles, using different Babel presets, for different browsers. Eg. for Chrome the only thing you need is module imports, but for Safari you use a preset that also includes destructuring, function defaults, etc. This reduces the attack surface that a rogue Babel dependency has access to - instead of being able to pwn your server and all browsers, it can only pwn the browsers that use the particular transform that depends on it. It also can result in significant byte-size savings in the downloaded bundle.
  - deleted 11y ago
    [deleted]
insin 11y ago
It's specifically a Babel 6 problem - the way plugins are published means it now implicitly requires npm3. Similarly, I had to bail on installing the 'victory' package (React visualisation components) using npm2 more than 600MB into it attempting to pull down its dependencies - I don't know if this is still the case, but every "victory-*" plugin was installing its own copies of Babel 6 and Webpack...