7 ms·

Yarn vs. npm5 determinism

72 points by styfle 9y ago

viraptor 9y ago
What's hoisting dependencies? I can't find a good description.
- hzoo 9y ago
  It's just about moving a dependency up a level of node_modules if they are compatible. So if for example 2 packages A and B have the same dependency on babel-core, instead of node_modules/A/node_modules/babel-core and node_modules/B/node_modules/babel-core you could just have node_modules/babel-core.
  - underyx 9y ago
    What's the benefit of this? Saving disk space?
    - meroje 9y ago
      Yes, and install speed too since you're writing the files once
      - avaer 9y ago
        Runtime speed also. Requiring only one copy means only one instance is parsed and optimized by the js engine. Which saves time and heats cache.
        gedy 9y ago
        Yes, but only if you aren't already transpiring minifying code such as for front end frameworks
        avaer 9y ago
        Which is most of the time, on the backend.
jazoom 9y ago
I may be missing something, but it sounds like the only benefit to Yarn is that the lock file is less noisy, while NPM has better guarantees. Who wants to deal with keeping absolutely every developer and every build/production system on exactly the same version of Yarn?
- horsawlarway 9y ago
  I mostly agree with you, but two points: 1. The amount of noise in the lock file likely does matter for teams of any modest size 2. Yarn doesn't guarantee determinism for hoisting between different versions, but... in reality there's no reason to expect different behavior simply because there's a version difference. The second point is pretty key, because it means yarn can deal with the issue through fairly standard means (like versioning, documentation, warnings, etc). Plus, most orgs already handle versioning and compatibility between tools in some form or other. Finally, NPM 5 doesn't actually guarantee hoisting is deterministic between versions, they just have enough information that they COULD guarantee it. For all we know, NPM 6 will change the format of the file entirely... or perform new hoisting optimizations based on the lockfile that are incompatible with the logic in 5.
naiveattack 9y ago
Where is the hoisting information in the npm lock file? The examples for yarn and npm look The same to me. Copied below: has-flag@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/has-flag/-/has-flag-1.0.0.tgz#9d9e793165ce017a00f00418c43f942a7b1d11fa" https://registry.yarnpkg.com/has-flag/-/has-flag-1.0.0.tgz#9... supports-color@^3.2.3: version "3.2.3" resolved "https://registry.yarnpkg.com/supports-color/-/supports-color-3.2.3.tgz#65ac0504b3954171d8a64946b2ae3cbb8a5f54f6" https://registry.yarnpkg.com/supports-color/-/supports-color... dependencies: has-flag "^1.0.0" { "name": "react-example", "version": "1.0.0", "lockfileVersion": 1, "dependencies": { "has-flag": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/has-flag/-/has-flag-1.0.0.tgz" https://registry.npmjs.org/has-flag/-/has-flag-1.0.0.tgz", "integrity": "sha1-nZ55MWXOAXoA8AQYxD+UKnsdEfo=" }, "supports-color": { "version": "3.2.3", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-3.2.3.tgz" https://registry.npmjs.org/supports-color/-/supports-color-3..., "integrity": "sha1-ZawFBLOVQXHYpklGsq48u4pfVPY=" } } } Yarn file seems to have more not less info.
nailer 9y ago
Can people please stop using 'lockfile' for things that aren't lockfiles? Particularly if you're storing important data using the name of a file type that, for the last few decades, has been something any Linux or Unix user can delete without significant consequence.
- sprt 9y ago
  Kenneth Reitz's pipenv and Go's future official package manager use that terminology as well, which doesn't make any sense.
- avaer 9y ago
  I have the same problem with "tree shaking". Granted it's not a bastardization of an established term, but it's a reinvention and confounding of terms we already had. I guess the one thing to take away from this is that js tool authors and c tool authors don't talk to each other enough.
  - tonyarkles 9y ago
    As a curiosity, and I say this as someone who has a significant C background, what would you call a similar concept in C? I know that it happens (having been burned by missing symbol errors due to incorrect link ordering), but I don't actually know what it's called.
    - bjterry 9y ago
      Typically it would be called dead-code elimination. The creator of rollup who popularized the term tree-shaking disagrees that they are the same, but his argument is not super compelling (not that it really matters either way). https://medium.com/@Rich_Harris/tree-shaking-versus-dead-code-elimination-d3765df85c80 https://medium.com/@Rich_Harris/tree-shaking-versus-dead-cod...
HereBeBeasties 9y ago
Unfortunately, despite years of bug reports, neither NPM nor the new kid on the block are deterministic when invoked on multiple builds on the same machine simultaneously. It's almost like their developers never use continuous integration environments. Ironically, if they used an actual lockfile in the traditional sense of the word and did that properly...
- cpojer 9y ago
  Yarn has a "--mutex" option for exactly this purpose.
fold_left 9y ago
Shrinkpack[1] allows npm to provide completely deterministic builds. [1] https://github.com/JamieMason/shrinkpack https://github.com/JamieMason/shrinkpack

72 points by styfle 9y ago

viraptor 9y ago

What's hoisting dependencies? I can't find a good description.

hzoo 9y ago
It's just about moving a dependency up a level of node_modules if they are compatible. So if for example 2 packages A and B have the same dependency on babel-core, instead of node_modules/A/node_modules/babel-core and node_modules/B/node_modules/babel-core you could just have node_modules/babel-core.
- underyx 9y ago
  What's the benefit of this? Saving disk space?
  - meroje 9y ago
    Yes, and install speed too since you're writing the files once
    - avaer 9y ago
      Runtime speed also. Requiring only one copy means only one instance is parsed and optimized by the js engine. Which saves time and heats cache.
      - gedy 9y ago
        Yes, but only if you aren't already transpiring minifying code such as for front end frameworks
        avaer 9y ago
        Which is most of the time, on the backend.

jazoom 9y ago

I may be missing something, but it sounds like the only benefit to Yarn is that the lock file is less noisy, while NPM has better guarantees. Who wants to deal with keeping absolutely every developer and every build/production system on exactly the same version of Yarn?

horsawlarway 9y ago
I mostly agree with you, but two points: 1. The amount of noise in the lock file likely does matter for teams of any modest size 2. Yarn doesn't guarantee determinism for hoisting between different versions, but... in reality there's no reason to expect different behavior simply because there's a version difference. The second point is pretty key, because it means yarn can deal with the issue through fairly standard means (like versioning, documentation, warnings, etc). Plus, most orgs already handle versioning and compatibility between tools in some form or other. Finally, NPM 5 doesn't actually guarantee hoisting is deterministic between versions, they just have enough information that they COULD guarantee it. For all we know, NPM 6 will change the format of the file entirely... or perform new hoisting optimizations based on the lockfile that are incompatible with the logic in 5.

naiveattack 9y ago

Where is the hoisting information in the npm lock file? The examples for yarn and npm look The same to me. Copied below: has-flag@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/has-flag/-/has-flag-1.0.0.tgz#9d9e793165ce017a00f00418c43f942a7b1d11fa" https://registry.yarnpkg.com/has-flag/-/has-flag-1.0.0.tgz#9... supports-color@^3.2.3: version "3.2.3" resolved "https://registry.yarnpkg.com/supports-color/-/supports-color-3.2.3.tgz#65ac0504b3954171d8a64946b2ae3cbb8a5f54f6" https://registry.yarnpkg.com/supports-color/-/supports-color... dependencies: has-flag "^1.0.0" { "name": "react-example", "version": "1.0.0", "lockfileVersion": 1, "dependencies": { "has-flag": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/has-flag/-/has-flag-1.0.0.tgz" https://registry.npmjs.org/has-flag/-/has-flag-1.0.0.tgz", "integrity": "sha1-nZ55MWXOAXoA8AQYxD+UKnsdEfo=" }, "supports-color": { "version": "3.2.3", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-3.2.3.tgz" https://registry.npmjs.org/supports-color/-/supports-color-3..., "integrity": "sha1-ZawFBLOVQXHYpklGsq48u4pfVPY=" } } } Yarn file seems to have more not less info.

nailer 9y ago

Can people please stop using 'lockfile' for things that aren't lockfiles? Particularly if you're storing important data using the name of a file type that, for the last few decades, has been something any Linux or Unix user can delete without significant consequence.

sprt 9y ago
Kenneth Reitz's pipenv and Go's future official package manager use that terminology as well, which doesn't make any sense.
avaer 9y ago
I have the same problem with "tree shaking". Granted it's not a bastardization of an established term, but it's a reinvention and confounding of terms we already had. I guess the one thing to take away from this is that js tool authors and c tool authors don't talk to each other enough.
- tonyarkles 9y ago
  As a curiosity, and I say this as someone who has a significant C background, what would you call a similar concept in C? I know that it happens (having been burned by missing symbol errors due to incorrect link ordering), but I don't actually know what it's called.
  - bjterry 9y ago
    Typically it would be called dead-code elimination. The creator of rollup who popularized the term tree-shaking disagrees that they are the same, but his argument is not super compelling (not that it really matters either way). https://medium.com/@Rich_Harris/tree-shaking-versus-dead-code-elimination-d3765df85c80 https://medium.com/@Rich_Harris/tree-shaking-versus-dead-cod...

HereBeBeasties 9y ago

Unfortunately, despite years of bug reports, neither NPM nor the new kid on the block are deterministic when invoked on multiple builds on the same machine simultaneously. It's almost like their developers never use continuous integration environments. Ironically, if they used an actual lockfile in the traditional sense of the word and did that properly...

cpojer 9y ago
Yarn has a "--mutex" option for exactly this purpose.

fold_left 9y ago

Shrinkpack[1] allows npm to provide completely deterministic builds. [1] https://github.com/JamieMason/shrinkpack https://github.com/JamieMason/shrinkpack