8 ms·

Yarn 1.0: Workspaces, auto-merging lockfiles, selective versions resolutions

262 points by cpojer 9y ago

nailer 9y ago
I've been holding off on yarn since npm v5 has some big performance increases, but there's (as of npm 5.3) still an issue where npm v5 deletes all `private: true` packages - we've had to revert to 4.6 - and I'm a little worried a data-loss issue isn't being treated as severe.
- allover 9y ago
  Deletes them in what situation? Can you provide a link?
  - nailer 9y ago
    There's a couple of similar deletion issues, they're all linked from https://github.com/npm/npm/issues/17929 https://github.com/npm/npm/issues/17929, the deleting private one happens in all situations, eg just an npm install, see https://github.com/npm/npm/issues/17927 https://github.com/npm/npm/issues/17927
    - Zekio 9y ago
      That explains why I when I use npm I have 24k files in my deleted files folder only for npm to fail.. after which I remember to use yarn
    - allover 9y ago
      RE the 2nd issue: Are you sure you even need `private: true`? Scoped modules are private by default right? I thought `private: true` was supposed to prevent publishing to npm altogether. It's odd they haven't at least triaged the issue (but you know humans/time). That said the fact it's a niche workflow (it's unclear why the user is using --no-save) and the steps to reproduce won't be doable by the team ... probably doesn't help the issue get attention.
      - nailer 9y ago
        The packages aren't published on npm but deployed by other means (before npm is used to install additional public packages), hence 'private' is deliberate.
        deleted 9y ago
        [deleted]
orta 9y ago
A big congrats are in order. Yarn came out just as I started working in the node ecosystem, and it fixed nearly all my dependency manager problems. Since then it's evolved and it's really helped provide a counter to npm offering different design choices but working together where it counts. It's the 4th most popular brew dependency and ~25% of npm downloads. People love it. Congrats Yarn team - you've been doing a great job. * https://stats.yarnpkg.com https://stats.yarnpkg.com * https://brew.sh/analytics/install-on-request/ https://brew.sh/analytics/install-on-request/
- brad0 9y ago
  I jumped on node around the same time and I've come to an opposite conclusion. The things I liked was the offline cache and the yarn integrity. However there was just too many issues regarding its dependency on npm and not playing nice with other tools such as brew. I dove into the code to try and fix these issues but both the quality of the code base and the politics going on in the issue tracker turned me off very quickly. [1] If yarn had been approached with more senior developers I think it could have replaced npm entirely. [1] https://github.com/yarnpkg/yarn/issues/2064 https://github.com/yarnpkg/yarn/issues/2064
  - deleted 9y ago
    [deleted]
  - petetnt 9y ago
    > If yarn had been approached with more senior developers I think it could have replaced npm entirely. The biggest reason by far why Yarn got so popular so quickly was because it didn't come and try to split the existing ecosystem but chose build on top of npm's foundation with ways that npm didn't (at that point) offer instead.
    - brad0 9y ago
      Yes I agree! I didn't really make myself clear. I feel that if yarn kept its strategy but had more senior developers writing the core software I think it could have been able to outpace npm in terms of features and speed. If this had have been the case then it wouldn't make sense to use npm at all. Yarn would be faster and have more features. From my experience I feel that yarn isn't offering much value now and the codebase quality means that future features will take too long to build.
adamgiacomelli 9y ago
I have had a failed heroku deployment because of yarn 1.0 yesterday - quite interesting. Made me define the yarn version in package.json. I have also had some problems with the yarn repo being unavailable a couple of months ago. That being said - I have migrated to it completely on all projects and never looked back - so yay for yarn!
- BYK 9y ago
  We have patched that issue and already tagged 1.0.1 ;)
  - BYK 9y ago
    See https://github.com/yarnpkg/yarn/issues/4320 https://github.com/yarnpkg/yarn/issues/4320
    - tarr11 9y ago
      Looks like its still having problems on Heroku https://github.com/heroku/heroku-buildpack-nodejs/issues/468 https://github.com/heroku/heroku-buildpack-nodejs/issues/468
- tarr11 9y ago
  My heroku deploy succeeded, but my CircleCI run failed for the same reason. Had to upgrade package.json as well, since my circleci build downloads yarn each time. Probably not the most efficient way of doing this but it worked...
WM6v 9y ago
How soon will it be available via brew update?
- arcatek 9y ago
  For the time being you should be able to install it using the `--devel` flag, but the PR to move it to stable is on its way :)
- deleted 9y ago
  [deleted]
- kshvmdn 9y ago
  Looks like 1.0.1 is available now - https://github.com/Homebrew/homebrew-core/pull/17780 https://github.com/Homebrew/homebrew-core/pull/17780.
nwlieb 9y ago
This is fantastic! The biggest standout feature in my opinion is workspaces. I've been using workspaces on a client project with many sub-projects and it has been a pleasure. Shared modularized code without creating private npm packages or doing some "linking" magic has been wonderful for productivity. It's as simple as creating another local package and symatically everything else has remained the same as a regular npm package, plus the benefits of having immediately updating code. For anyone with a large modular codebase wanting to forray into a monorepo approach I highly recommend checking it out. They also released a blog post here detailing the feature: https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/ https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/
- arcatek 9y ago
  We've also published a more detailed documentation here: https://yarnpkg.com/en/docs/workspaces https://yarnpkg.com/en/docs/workspaces!
- Rapzid 9y ago
  How are people handling CI/CD servers/services with the mono repo? The barrier to mono repo seems to have always been the need to create massive amounts of bespoke tooling to handle the build pipelines(how do I build just this proj?, how do I run tests on just this proj?, how do I build just this proj and its deps?, etc). These are totally solvable with dedicated dev resources, but these are typically the problems you try to avoid rather than spend time on at a lot of companies; say < 1k engineers(snark). Workspaces and Lerna seem to be a small piece of the puzzle even if critical. Are there tools to help out with rest? For instance if I'm using Bitbucket pipelines, what should I be looking for to help building from a mono repo?
  - hinkley 9y ago
    Subversion used to let you check out a subdirectory of a repo, which generally helped with this problem. The chapter on SVN in Beautiful Code does a great job of explaining how this works and why it helps with concurrent access to the repo. This is a giant blind spot in Git, and none of the proposed workarounds come anywhere close to the tiny cognitive load of svn's answer to this problem. But nobody on the git design committee seems to give a shit about cognitive load, so I'm probably just shouting into the tempest.
noir_lord 9y ago
Yarn has been a collosal improvement to my workflow, it's faster, compatible with non and completely deterministic (those weird this install works that one didn't errors with npm got really old).
msoad 9y ago
Maybe Node should bundle Yarn instead of NPM? NPM is so buggy I can't stand it for a minute!
- scarlac 9y ago
  Not quite the same but ironically the standard "node" Docker image contains yarn: https://github.com/nodejs/docker-node/blob/b502aa016335c81a586b430328d8fee4897ee440/8.4/Dockerfile https://github.com/nodejs/docker-node/blob/b502aa016335c81a5...
  - onestone 9y ago
    I was the one who added Yarn to the official docker-node image[1][2]. This was the most popular docker-node feature request at the time. We managed to achieve it with a minimal size increase (with the help of the awesome Yarn team) and without affecting anything else. There was already a history of us providing some bonus features which the upstream Node project doesn't, such as a musl-libc build (for Alpine Linux). [1] https://github.com/nodejs/docker-node/pull/337 https://github.com/nodejs/docker-node/pull/337 [2] https://github.com/nodejs/docker-node/pull/403 https://github.com/nodejs/docker-node/pull/403
marricks 9y ago
What was the compelling reason to make Yarn instead of contributing to npm, perhaps ownership? Seems like they could given everyone a lot of improvements and instead they fractured the market a bit.
- sotojuan 9y ago
  npm code is a mess and contributing to it is a very slow process. Competition from yarn led to npm quickly adopting lockfiles and being faster.
- brentvatne 9y ago
  I can't speak to the original intentions of the authors, but I suspect it might be related to difficulty of making significant changes to the project due to some strongly held beliefs. An example: react-native depends on alpha/beta versions of react, and libraries in the react-native ecosystem tend to include a peer dependency on react (eg: react >= 15). React 16.0.0-alpha.12 will not satisfy this range, but 16.0.0 will. It's unclear to me in what way it is useful to exclude pre-release versions, and this causes a lot of confusion for users. I posted about it here: https://github.com/npm/npm/issues/8854 https://github.com/npm/npm/issues/8854 and it was shutdown for ideological reasons, rather than practical considerations. On the other hand, someone submitted a patch to yarn to improve this behaviour and it was quickly accepted: https://github.com/yarnpkg/yarn/pull/3361 https://github.com/yarnpkg/yarn/pull/3361.
  - Klathmon 9y ago
    And it wasn't all just "strongly held beliefs" there were very real problems with upgrading parts of npm. npm for a long time was built to do what npm did. There was no spec, no "rules" it did what it did, and changing that behavior was a breaking change. Everyone could agree that feature X needed to be fixed or redone, but doing so would break a significant number of packages/projects so it wasn't done. Yarn was the solution, they could start from scratch, not worry about those older/undocumented/arcane edge-cases. They didn't have to care about backwards compatibility, or even reimplement the same API. Every time this question is asked, I also like to point out that many of the Yarn devs were npm devs, and the project as a whole not only had the "approval" of npm, but also was in-part encouraged by them. Competition is good, and it's "the javascript way" to have multiple competing tools that each prioritise different things. Yarn is pretty much a perfect example of that.
firloop 9y ago
Will Yarn ever be merged into npm's CLI? Why does the node community back two package managers?
- tengkahwee 9y ago
  Node has more than 2 package managers. There's also jspm and at some point of them people were using bower too.
- Klathmon 9y ago
  The same reason most people back multiple browser rendering engines, or multiple car manufacturers, or multiple CPU manufacturers. Competition is good, and having 2 implementations of the same basic idea that each prioritise different things means that more people can have their cake and eat it too. Having one tool that has hundreds of flags to enable different use-cases is an anti-pattern in my opinion, and a fantastic alternative is to create multiple competing tools that each handle those different use-cases. It's the unix philosophy taken to the extreme, each "program" should do one thing, and do it well.
  - firloop 9y ago
    This is a good answer that changed my perspective, thanks.
williamle8300 9y ago
Do they use the same BSD+Patents license that's used for ReactJS?
- talkingtab 9y ago
  The license on github does not have the patents clause.
MikeTaylor 9y ago
Just in case anyone else runs into this and doesn't know what's going on: we had a problem with Yarn 1.0, because it doesn't recognise repositories that don't end in a trailing slash -- for example, "@folio:registry=https://repository.folio.org/repository/npm-folio" https://repository.folio.org/repository/npm-folio" is no good any more, but "@folio:registry=https://repository.folio.org/repository/npm-folio/" https://repository.folio.org/repository/npm-folio/" is. See https://github.com/folio-org/stripes-demo-platform/commit/f8fed68d17c72b021a6c92ebe74fc173a7ea0433 https://github.com/folio-org/stripes-demo-platform/commit/f8... Meanwhile ... I do regret that this doesn't introduce alternative dependencies, like Debian packages have, where your package foo can have dependency on "either bar or baz". Oh well.
- BYK 9y ago
  Apologies for the issue. There's an issue file on GitHub for this and we are working on a fix: https://github.com/yarnpkg/yarn/issues/4339 https://github.com/yarnpkg/yarn/issues/4339

262 points by cpojer 9y ago

I've been holding off on yarn since npm v5 has some big performance increases, but there's (as of npm 5.3) still an issue where npm v5 deletes all `private: true` packages - we've had to revert to 4.6 - and I'm a little worried a data-loss issue isn't being treated as severe.

allover 9y ago
Deletes them in what situation? Can you provide a link?
- nailer 9y ago
  There's a couple of similar deletion issues, they're all linked from https://github.com/npm/npm/issues/17929 https://github.com/npm/npm/issues/17929, the deleting private one happens in all situations, eg just an npm install, see https://github.com/npm/npm/issues/17927 https://github.com/npm/npm/issues/17927
  - Zekio 9y ago
    That explains why I when I use npm I have 24k files in my deleted files folder only for npm to fail.. after which I remember to use yarn
  - allover 9y ago
    RE the 2nd issue: Are you sure you even need `private: true`? Scoped modules are private by default right? I thought `private: true` was supposed to prevent publishing to npm altogether. It's odd they haven't at least triaged the issue (but you know humans/time). That said the fact it's a niche workflow (it's unclear why the user is using --no-save) and the steps to reproduce won't be doable by the team ... probably doesn't help the issue get attention.
    - nailer 9y ago
      The packages aren't published on npm but deployed by other means (before npm is used to install additional public packages), hence 'private' is deliberate.
      - deleted 9y ago
        [deleted]

orta 9y ago

A big congrats are in order. Yarn came out just as I started working in the node ecosystem, and it fixed nearly all my dependency manager problems. Since then it's evolved and it's really helped provide a counter to npm offering different design choices but working together where it counts. It's the 4th most popular brew dependency and ~25% of npm downloads. People love it. Congrats Yarn team - you've been doing a great job. * https://stats.yarnpkg.com https://stats.yarnpkg.com * https://brew.sh/analytics/install-on-request/ https://brew.sh/analytics/install-on-request/

brad0 9y ago
I jumped on node around the same time and I've come to an opposite conclusion. The things I liked was the offline cache and the yarn integrity. However there was just too many issues regarding its dependency on npm and not playing nice with other tools such as brew. I dove into the code to try and fix these issues but both the quality of the code base and the politics going on in the issue tracker turned me off very quickly. [1] If yarn had been approached with more senior developers I think it could have replaced npm entirely. [1] https://github.com/yarnpkg/yarn/issues/2064 https://github.com/yarnpkg/yarn/issues/2064
- deleted 9y ago
  [deleted]
- petetnt 9y ago
  > If yarn had been approached with more senior developers I think it could have replaced npm entirely. The biggest reason by far why Yarn got so popular so quickly was because it didn't come and try to split the existing ecosystem but chose build on top of npm's foundation with ways that npm didn't (at that point) offer instead.
  - brad0 9y ago
    Yes I agree! I didn't really make myself clear. I feel that if yarn kept its strategy but had more senior developers writing the core software I think it could have been able to outpace npm in terms of features and speed. If this had have been the case then it wouldn't make sense to use npm at all. Yarn would be faster and have more features. From my experience I feel that yarn isn't offering much value now and the codebase quality means that future features will take too long to build.

adamgiacomelli 9y ago

I have had a failed heroku deployment because of yarn 1.0 yesterday - quite interesting. Made me define the yarn version in package.json. I have also had some problems with the yarn repo being unavailable a couple of months ago. That being said - I have migrated to it completely on all projects and never looked back - so yay for yarn!

BYK 9y ago
We have patched that issue and already tagged 1.0.1 ;)
- BYK 9y ago
  See https://github.com/yarnpkg/yarn/issues/4320 https://github.com/yarnpkg/yarn/issues/4320
  - tarr11 9y ago
    Looks like its still having problems on Heroku https://github.com/heroku/heroku-buildpack-nodejs/issues/468 https://github.com/heroku/heroku-buildpack-nodejs/issues/468
tarr11 9y ago
My heroku deploy succeeded, but my CircleCI run failed for the same reason. Had to upgrade package.json as well, since my circleci build downloads yarn each time. Probably not the most efficient way of doing this but it worked...

WM6v 9y ago

How soon will it be available via brew update?

arcatek 9y ago
For the time being you should be able to install it using the `--devel` flag, but the PR to move it to stable is on its way :)
deleted 9y ago
[deleted]
kshvmdn 9y ago
Looks like 1.0.1 is available now - https://github.com/Homebrew/homebrew-core/pull/17780 https://github.com/Homebrew/homebrew-core/pull/17780.

nwlieb 9y ago

This is fantastic! The biggest standout feature in my opinion is workspaces. I've been using workspaces on a client project with many sub-projects and it has been a pleasure. Shared modularized code without creating private npm packages or doing some "linking" magic has been wonderful for productivity. It's as simple as creating another local package and symatically everything else has remained the same as a regular npm package, plus the benefits of having immediately updating code. For anyone with a large modular codebase wanting to forray into a monorepo approach I highly recommend checking it out. They also released a blog post here detailing the feature: https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/ https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/

arcatek 9y ago
We've also published a more detailed documentation here: https://yarnpkg.com/en/docs/workspaces https://yarnpkg.com/en/docs/workspaces!
Rapzid 9y ago
How are people handling CI/CD servers/services with the mono repo? The barrier to mono repo seems to have always been the need to create massive amounts of bespoke tooling to handle the build pipelines(how do I build just this proj?, how do I run tests on just this proj?, how do I build just this proj and its deps?, etc). These are totally solvable with dedicated dev resources, but these are typically the problems you try to avoid rather than spend time on at a lot of companies; say < 1k engineers(snark). Workspaces and Lerna seem to be a small piece of the puzzle even if critical. Are there tools to help out with rest? For instance if I'm using Bitbucket pipelines, what should I be looking for to help building from a mono repo?
- hinkley 9y ago
  Subversion used to let you check out a subdirectory of a repo, which generally helped with this problem. The chapter on SVN in Beautiful Code does a great job of explaining how this works and why it helps with concurrent access to the repo. This is a giant blind spot in Git, and none of the proposed workarounds come anywhere close to the tiny cognitive load of svn's answer to this problem. But nobody on the git design committee seems to give a shit about cognitive load, so I'm probably just shouting into the tempest.

noir_lord 9y ago

Yarn has been a collosal improvement to my workflow, it's faster, compatible with non and completely deterministic (those weird this install works that one didn't errors with npm got really old).

msoad 9y ago

Maybe Node should bundle Yarn instead of NPM? NPM is so buggy I can't stand it for a minute!

scarlac 9y ago
Not quite the same but ironically the standard "node" Docker image contains yarn: https://github.com/nodejs/docker-node/blob/b502aa016335c81a586b430328d8fee4897ee440/8.4/Dockerfile https://github.com/nodejs/docker-node/blob/b502aa016335c81a5...
- onestone 9y ago
  I was the one who added Yarn to the official docker-node image[1][2]. This was the most popular docker-node feature request at the time. We managed to achieve it with a minimal size increase (with the help of the awesome Yarn team) and without affecting anything else. There was already a history of us providing some bonus features which the upstream Node project doesn't, such as a musl-libc build (for Alpine Linux). [1] https://github.com/nodejs/docker-node/pull/337 https://github.com/nodejs/docker-node/pull/337 [2] https://github.com/nodejs/docker-node/pull/403 https://github.com/nodejs/docker-node/pull/403

marricks 9y ago

What was the compelling reason to make Yarn instead of contributing to npm, perhaps ownership? Seems like they could given everyone a lot of improvements and instead they fractured the market a bit.

sotojuan 9y ago
npm code is a mess and contributing to it is a very slow process. Competition from yarn led to npm quickly adopting lockfiles and being faster.
brentvatne 9y ago
I can't speak to the original intentions of the authors, but I suspect it might be related to difficulty of making significant changes to the project due to some strongly held beliefs. An example: react-native depends on alpha/beta versions of react, and libraries in the react-native ecosystem tend to include a peer dependency on react (eg: react >= 15). React 16.0.0-alpha.12 will not satisfy this range, but 16.0.0 will. It's unclear to me in what way it is useful to exclude pre-release versions, and this causes a lot of confusion for users. I posted about it here: https://github.com/npm/npm/issues/8854 https://github.com/npm/npm/issues/8854 and it was shutdown for ideological reasons, rather than practical considerations. On the other hand, someone submitted a patch to yarn to improve this behaviour and it was quickly accepted: https://github.com/yarnpkg/yarn/pull/3361 https://github.com/yarnpkg/yarn/pull/3361.
- Klathmon 9y ago
  And it wasn't all just "strongly held beliefs" there were very real problems with upgrading parts of npm. npm for a long time was built to do what npm did. There was no spec, no "rules" it did what it did, and changing that behavior was a breaking change. Everyone could agree that feature X needed to be fixed or redone, but doing so would break a significant number of packages/projects so it wasn't done. Yarn was the solution, they could start from scratch, not worry about those older/undocumented/arcane edge-cases. They didn't have to care about backwards compatibility, or even reimplement the same API. Every time this question is asked, I also like to point out that many of the Yarn devs were npm devs, and the project as a whole not only had the "approval" of npm, but also was in-part encouraged by them. Competition is good, and it's "the javascript way" to have multiple competing tools that each prioritise different things. Yarn is pretty much a perfect example of that.

firloop 9y ago

Will Yarn ever be merged into npm's CLI? Why does the node community back two package managers?

tengkahwee 9y ago
Node has more than 2 package managers. There's also jspm and at some point of them people were using bower too.
Klathmon 9y ago
The same reason most people back multiple browser rendering engines, or multiple car manufacturers, or multiple CPU manufacturers. Competition is good, and having 2 implementations of the same basic idea that each prioritise different things means that more people can have their cake and eat it too. Having one tool that has hundreds of flags to enable different use-cases is an anti-pattern in my opinion, and a fantastic alternative is to create multiple competing tools that each handle those different use-cases. It's the unix philosophy taken to the extreme, each "program" should do one thing, and do it well.
- firloop 9y ago
  This is a good answer that changed my perspective, thanks.

williamle8300 9y ago

Do they use the same BSD+Patents license that's used for ReactJS?

talkingtab 9y ago
The license on github does not have the patents clause.

MikeTaylor 9y ago

Just in case anyone else runs into this and doesn't know what's going on: we had a problem with Yarn 1.0, because it doesn't recognise repositories that don't end in a trailing slash -- for example, "@folio:registry=https://repository.folio.org/repository/npm-folio" https://repository.folio.org/repository/npm-folio" is no good any more, but "@folio:registry=https://repository.folio.org/repository/npm-folio/" https://repository.folio.org/repository/npm-folio/" is. See https://github.com/folio-org/stripes-demo-platform/commit/f8fed68d17c72b021a6c92ebe74fc173a7ea0433 https://github.com/folio-org/stripes-demo-platform/commit/f8... Meanwhile ... I do regret that this doesn't introduce alternative dependencies, like Debian packages have, where your package foo can have dependency on "either bar or baz". Oh well.

BYK 9y ago
Apologies for the issue. There's an issue file on GitHub for this and we are working on a fix: https://github.com/yarnpkg/yarn/issues/4339 https://github.com/yarnpkg/yarn/issues/4339