6 ms·

First of all: Thank you, yarn, for helping the community see the naked emperor. Deterministic builds by default are such an obvious (in retrospect) core requi

First of all: Thank you, yarn, for helping the community see the naked emperor. Deterministic builds by default are such an obvious (in retrospect) core requirement. Couple questions: Question 1: Does anyone else who's been around more than a couple years share my view that Yarn : NPM :: IO.JS : Node? IOW: healthy competition, catalyst for necessary change, ultimately a bridge or stopgap. Question 2: Any good comprehensive writeups on best practices?Committing lockfiles is a no-brainer. But what about globals? Per-node-version seems logical, but there are also semantic problems with "global" packages vs concept of executable binaries. I'd love to see a strong writeup outlining and defending a standard approach.

pluma 9y ago
When io.js merged back into node.js the following happened: 1. Node.js was abandoned and replaced with io.js. 2. Io.js was relabelled as node.js. 3. The node.js project was put under open governance via the creation of the Node Foundation. This doesn't really compare to npm: * The npm-cli's name is using npm Inc's trademark: giving it away would leave the company with no name or create unwanted ambiguity between npm Inc the company and npm-cli the (then) community-owned open source project unaffiliated with the company. * Yarn does not share any source code directly with npm-cli, it's a complete reimplementation of a similar feature set. It's currently backed by a mirror of the npm registry but that's the extent of the overlap. * The Node Foundation already exists and Yarn is already owned by the community. I could see Yarn joining the Node Foundation and the Node Foundation replacing its bundled dependency on the commercially owned npm-cli but even then npm Inc has nothing to contribute to the process. Personally I would much rather see Yarn join the JavaScript Foundation because it speaks to the broader appeal of Yarn outside the traditional "Node community" and the politics involved in the latter and its close ties to npm Inc. FWIW I consider the close ties between the Node project and npm Inc nothing more than a historical accident that has resulted in a conflict of interest that is overdue to be resolved by dropping npm-cli from the official releases. Now that yarn is no longer distributed using npm (although it's still possible to install it that way) that seems more realistic than ever.
- ultim8k 9y ago
  They probably need to do something like docker did.
- cheapsteak 9y ago
  >resolved by dropping npm-cli from the official releases. But that would break 7 years of documentation and tutorials, likely bad for newcomers and thus the ecosystem
  - josteink 9y ago
    Not to mention quite a few CI build pipelines.
    - pluma 9y ago
      FWIW it's trivial to install yarn already: https://yarnpkg.com/en/docs/install https://yarnpkg.com/en/docs/install NPM currently doesn't provide a standalone distribution as far as I can tell, but presumably they would offer one if they no longer had the luxury of being bundled with Node. Since changes generally don't happen overnight I would expect a transitional period where Node still bundles npm-cli but npm Inc has the time to prepare a standalone distribution before yarn replaces npm-cli. Additionally downstream channels could decide to provide legacy packages containing both.
  - k__ 9y ago
    so npm has created a huge vendor lock-in and we should live with it?!
    - cheapsteak 9y ago
      As with any vendor lock-in situation, decisions should be made based on cost/benefit Perhaps you can help come up with a strategy that gracefully handles breakages and eventually results in dropping npm-cli
    - SilasX 9y ago
      "The only reason God could create the world in six days is, He didn't have to worry about backward compatibility."
nailer 9y ago
> Thank you, yarn, for helping the community see the naked emperor. Deterministic builds by default are such an obvious (in retrospect) core requirement. https://docs.npmjs.com/cli/shrinkwrap https://docs.npmjs.com/cli/shrinkwrap provides deterministic builds and has been around far longer than yarn. Since Oct 2014 npm v3 started automatically updating shrinkwrap whenever '--save' was used. See https://github.com/npm/npm/pull/4918#issuecomment-61344871 https://github.com/npm/npm/pull/4918#issuecomment-61344871 Calling npm 'the naked emperor' for not implementing something it did, infact, implement is uncalled for. Edit: added links in response to unexplained downmods. The parent is simply, provably wrong.
- bryanlarsen 9y ago
  Even with shrinkwrap, npm install is not deterministic, install order still matters. To get a deterministic install, you need to use shrinkwrap and do an `rm -rf node_modules` before every install. And it's still not completely deterministic if any dependency or sub-dependency has optionalDependencies. https://docs.npmjs.com/how-npm-works/npm3-nondet https://docs.npmjs.com/how-npm-works/npm3-nondet
  - nailer 9y ago
    I've read the https://docs.npmjs.com/how-npm-works/npm3-nondet https://docs.npmjs.com/how-npm-works/npm3-nondet but it make no mention of shrinkwrap. Since shrinkwrap captures versions for the entire tree, install order wouldn't matter.
  - dvlsg 9y ago
    That, and the fact that it's not the default behavior in npm gives yarn a pretty big advantage in my opinion. Especially for devs who are newer to Node.js.
- Silhouette 9y ago
  Shrinkwrap is a partially successful attempt to solve the problem, but still has obvious flaws. For example, instead of the relatively neat version specifications normally used in package.json, with a shrinkwrapped project you wind up with https://registry.npmjs.org/.. https://registry.npmjs.org/... paths instead, but only sometimes. This gets messy if you want to both install stable versions of all your packages and use a local registry to supply them for reliability, which is not an unlikely combination.
spankalee 9y ago
npm would need to add the --flat option for it to work for me
tlrobinson 9y ago
Answer 1: Yes but they're not exactly analogous and I don't see them merging. Yarn is a complete rewrite, and the compatibility "surface area" is much lower than Node itself so there's not much pressure for them to merge. Yarn has been working essentially flawlessly for me, so npm-cli is really going to have to surpass yarn in some way for me to switch back. Answer 2: I don't use global packages often. For binaries I generally add a "scripts" entry to package.json for common tasks, which automatically adds "./node_modules/.bin" to the PATH. For one-off tasks I just run "./node_modules/.bin/executable"
- Akkuma 9y ago
  Git dependencies with semver support seems to pretty much leapfrog yarn. One of the few reasons for a private npm package is removed with this feature.
rattray 9y ago
As @pluma mentioned, a "merge" seems unlikely (for one thing they don't share much, if any, code). However, npm could certainly compete with more of yarn's features (fast local cache[0], shorter syntax for running scripts like `npm start`, etc) [0] turns out they did that: https://twitter.com/maybekatz/status/865393382260056064 https://twitter.com/maybekatz/status/865393382260056064 wow!
draw_down 9y ago
Yeah, I think that's a good comparison, except the upsides of yarn seemed much more obvious to the community than was the case with IOJS. There seemed a lot less controversy than with IOJS. What a time those days were, feels like a million years ago already.
cphoover 9y ago
"Deterministic builds by default are such an obvious (in retrospect) core requirement." I agree... I like the theoretical concept that our dependencies, could be automatically managed/upgraded behind the scenes via a declarative versioning system, like semver. That, in my view, was the paradigm behind not having defaulted to dependency graph snapshot (lock/shrinkwrap) by default. In practice, people don't follow semver correctly and autoupdating patch+minor versions can cause you a lot of pain. It would be cool to make versioning more tied to actual API signature, so that changes to the signature could not actually be published as minor/patch changes... but would be forced to use a major version bump in publish. I don't know how this would work particularly in a language like javacscript w/out a typed interface... but I'm just theorizing...
IsaacSchlueter 9y ago
It's more accurate to say that npm v5 : npm :: io.js : node. Same team, performing a big refactoring and significant improvement and modernization of some bits that were outdated and difficult to improve without a controlled demolition and rebuild. The analogy breaks down, of course, because the governance didn't change, the project's relationship to its corporate backer wasn't standing in the way of progress, etc. (Except, I guess, that it might have gone faster if we'd had more money to hire more devs? But some of this was just a slog through a very old codebase with a lot of scar tissue, and it's hard to speed that up by putting more hands on it.) Everything in npm 5 was literally planned years in advance. When we have this many people depending on a thing, we have to be careful about how we make drastic changes. Yarn was a strong signal from the community that we were on the right track, but it only seems like a "catalyst" when seen from the outside. Correlation is not causation, even when it lands first.
- chrisweekly 9y ago
  Thanks Isaac! Both for this insight and - more so - for your massive contributions to the community. Slainte! :)

pluma 9y ago

When io.js merged back into node.js the following happened: 1. Node.js was abandoned and replaced with io.js. 2. Io.js was relabelled as node.js. 3. The node.js project was put under open governance via the creation of the Node Foundation. This doesn't really compare to npm: * The npm-cli's name is using npm Inc's trademark: giving it away would leave the company with no name or create unwanted ambiguity between npm Inc the company and npm-cli the (then) community-owned open source project unaffiliated with the company. * Yarn does not share any source code directly with npm-cli, it's a complete reimplementation of a similar feature set. It's currently backed by a mirror of the npm registry but that's the extent of the overlap. * The Node Foundation already exists and Yarn is already owned by the community. I could see Yarn joining the Node Foundation and the Node Foundation replacing its bundled dependency on the commercially owned npm-cli but even then npm Inc has nothing to contribute to the process. Personally I would much rather see Yarn join the JavaScript Foundation because it speaks to the broader appeal of Yarn outside the traditional "Node community" and the politics involved in the latter and its close ties to npm Inc. FWIW I consider the close ties between the Node project and npm Inc nothing more than a historical accident that has resulted in a conflict of interest that is overdue to be resolved by dropping npm-cli from the official releases. Now that yarn is no longer distributed using npm (although it's still possible to install it that way) that seems more realistic than ever.

ultim8k 9y ago
They probably need to do something like docker did.
cheapsteak 9y ago
>resolved by dropping npm-cli from the official releases. But that would break 7 years of documentation and tutorials, likely bad for newcomers and thus the ecosystem
- josteink 9y ago
  Not to mention quite a few CI build pipelines.
  - pluma 9y ago
    FWIW it's trivial to install yarn already: https://yarnpkg.com/en/docs/install https://yarnpkg.com/en/docs/install NPM currently doesn't provide a standalone distribution as far as I can tell, but presumably they would offer one if they no longer had the luxury of being bundled with Node. Since changes generally don't happen overnight I would expect a transitional period where Node still bundles npm-cli but npm Inc has the time to prepare a standalone distribution before yarn replaces npm-cli. Additionally downstream channels could decide to provide legacy packages containing both.
- k__ 9y ago
  so npm has created a huge vendor lock-in and we should live with it?!
  - cheapsteak 9y ago
    As with any vendor lock-in situation, decisions should be made based on cost/benefit Perhaps you can help come up with a strategy that gracefully handles breakages and eventually results in dropping npm-cli
  - SilasX 9y ago
    "The only reason God could create the world in six days is, He didn't have to worry about backward compatibility."

nailer 9y ago

> Thank you, yarn, for helping the community see the naked emperor. Deterministic builds by default are such an obvious (in retrospect) core requirement. https://docs.npmjs.com/cli/shrinkwrap https://docs.npmjs.com/cli/shrinkwrap provides deterministic builds and has been around far longer than yarn. Since Oct 2014 npm v3 started automatically updating shrinkwrap whenever '--save' was used. See https://github.com/npm/npm/pull/4918#issuecomment-61344871 https://github.com/npm/npm/pull/4918#issuecomment-61344871 Calling npm 'the naked emperor' for not implementing something it did, infact, implement is uncalled for. Edit: added links in response to unexplained downmods. The parent is simply, provably wrong.

bryanlarsen 9y ago
Even with shrinkwrap, npm install is not deterministic, install order still matters. To get a deterministic install, you need to use shrinkwrap and do an `rm -rf node_modules` before every install. And it's still not completely deterministic if any dependency or sub-dependency has optionalDependencies. https://docs.npmjs.com/how-npm-works/npm3-nondet https://docs.npmjs.com/how-npm-works/npm3-nondet
- nailer 9y ago
  I've read the https://docs.npmjs.com/how-npm-works/npm3-nondet https://docs.npmjs.com/how-npm-works/npm3-nondet but it make no mention of shrinkwrap. Since shrinkwrap captures versions for the entire tree, install order wouldn't matter.
- dvlsg 9y ago
  That, and the fact that it's not the default behavior in npm gives yarn a pretty big advantage in my opinion. Especially for devs who are newer to Node.js.
Silhouette 9y ago
Shrinkwrap is a partially successful attempt to solve the problem, but still has obvious flaws. For example, instead of the relatively neat version specifications normally used in package.json, with a shrinkwrapped project you wind up with https://registry.npmjs.org/.. https://registry.npmjs.org/... paths instead, but only sometimes. This gets messy if you want to both install stable versions of all your packages and use a local registry to supply them for reliability, which is not an unlikely combination.

spankalee 9y ago

npm would need to add the --flat option for it to work for me

tlrobinson 9y ago

Answer 1: Yes but they're not exactly analogous and I don't see them merging. Yarn is a complete rewrite, and the compatibility "surface area" is much lower than Node itself so there's not much pressure for them to merge. Yarn has been working essentially flawlessly for me, so npm-cli is really going to have to surpass yarn in some way for me to switch back. Answer 2: I don't use global packages often. For binaries I generally add a "scripts" entry to package.json for common tasks, which automatically adds "./node_modules/.bin" to the PATH. For one-off tasks I just run "./node_modules/.bin/executable"

Akkuma 9y ago
Git dependencies with semver support seems to pretty much leapfrog yarn. One of the few reasons for a private npm package is removed with this feature.

rattray 9y ago

As @pluma mentioned, a "merge" seems unlikely (for one thing they don't share much, if any, code). However, npm could certainly compete with more of yarn's features (fast local cache[0], shorter syntax for running scripts like `npm start`, etc) [0] turns out they did that: https://twitter.com/maybekatz/status/865393382260056064 https://twitter.com/maybekatz/status/865393382260056064 wow!

draw_down 9y ago

Yeah, I think that's a good comparison, except the upsides of yarn seemed much more obvious to the community than was the case with IOJS. There seemed a lot less controversy than with IOJS. What a time those days were, feels like a million years ago already.

cphoover 9y ago

"Deterministic builds by default are such an obvious (in retrospect) core requirement." I agree... I like the theoretical concept that our dependencies, could be automatically managed/upgraded behind the scenes via a declarative versioning system, like semver. That, in my view, was the paradigm behind not having defaulted to dependency graph snapshot (lock/shrinkwrap) by default. In practice, people don't follow semver correctly and autoupdating patch+minor versions can cause you a lot of pain. It would be cool to make versioning more tied to actual API signature, so that changes to the signature could not actually be published as minor/patch changes... but would be forced to use a major version bump in publish. I don't know how this would work particularly in a language like javacscript w/out a typed interface... but I'm just theorizing...

IsaacSchlueter 9y ago

It's more accurate to say that npm v5 : npm :: io.js : node. Same team, performing a big refactoring and significant improvement and modernization of some bits that were outdated and difficult to improve without a controlled demolition and rebuild. The analogy breaks down, of course, because the governance didn't change, the project's relationship to its corporate backer wasn't standing in the way of progress, etc. (Except, I guess, that it might have gone faster if we'd had more money to hire more devs? But some of this was just a slog through a very old codebase with a lot of scar tissue, and it's hard to speed that up by putting more hands on it.) Everything in npm 5 was literally planned years in advance. When we have this many people depending on a thing, we have to be careful about how we make drastic changes. Yarn was a strong signal from the community that we were on the right track, but it only seems like a "catalyst" when seen from the outside. Correlation is not causation, even when it lands first.

chrisweekly 9y ago
Thanks Isaac! Both for this insight and - more so - for your massive contributions to the community. Slainte! :)