34 ms·

pnpm: Fast, disk space efficient package manager for JavaScript

206 points by modinfo 5y ago

PeledYuval 5y ago
pnpm has been my goto JS monorepo package manager + script runner for a couple of years now. IMO it has almost zero downsides and huge upsides. I'd be surprised if npm doesn't end up adopting the pnpm node_modules directory hierarchy in the next 5 years or so.
- no_wizard 5y ago
  I believe the RFC for just that has landed: https://github.com/npm/rfcs/blob/main/accepted/0042-isolated-mode.md https://github.com/npm/rfcs/blob/main/accepted/0042-isolated... Yarn has a PNPM mode now too: https://dev.to/arcanis/yarn-31-corepack-esm-pnpm-optional-packages--3hak https://dev.to/arcanis/yarn-31-corepack-esm-pnpm-optional-pa...
synergy20 5y ago
This is my default one, _much_ faster and disk space reduced dramatically when I have lots of node_modules in use. Can't recommend enough.
- eatonphil 5y ago
  Were you coming from yarn or npm? I guess their benchmarks cover both [0]. But I'm also curious about independent figures. [0] https://pnpm.io/benchmarks https://pnpm.io/benchmarks
  - synergy20 5y ago
    used to use npm, less often with yarn, now it's all pnpm
    - eatonphil 5y ago
      Gotcha. Yeah yarn has always been way faster to me than npm. And indeed in these benchmarks pnpm is much faster than npm and only slightly faster than yarn.
CSDude 5y ago
It works very fast in CI, its cache is smaller and it builds node_modules much faster. I don't feel comfortable caching node_modules folder itself, because I got side effects even causing incidents before, not sure if its supposed to. Speed difference is 15s vs 35s for our use case which is pretty significant.
hu3 5y ago
Is there any gotcha if I switch from npm to pnpm? And can I use pnpm when working in a team where other devs use npm?
- jakubmazanec 5y ago
  Pnpm doesn't auto install peer dependencies, which is annoying and forces you to unnecessary add them to package.json. Npm@7 (and newer) with auto install of peer deps is much easier to use in this regard.
  - ricardobeat 5y ago
    Isn't that exactly how it's supposed to be? If you're installing a package that has peer dependencies, you should already depend on them. Otherwise they are not "peer dependencies" anymore, just a normal dependency with a version range. The whole peer dependencies story is another clusterfuck. Everyone simply ignored the invalid peer dependency warnings, and now npm itself will just install all of them in 'whatever works' version. To get real peer dependency resolution you need a --strict-peer-deps flag. Not exactly a "feature" in my book.
    - Wronnay 5y ago
      Yeah - this peer dependency thing always makes issues, when I try to upgrade a big Ionic project... Peer dependencies added more problems than they solved in my eyes... Most peer dependencies warnings come because some maintainer forgot to update the package.json and not really because two packages don't work with each other...
      - eyelidlessness 5y ago
        The point of peer dependencies is to allow users more flexibility in choosing the specific version of the dependency. It’s not intended to specify any kind of conflict between the packages, it’s effectively BYO sub-dependency.
yewenjie 5y ago
pnpm is awesome except when it does not work - which happens very rarely but it is a nightmare to debug. For all the other times, it is way faster and lightweight than npm.
- scambier 5y ago
  I was unable to use pnpm with a project that used Electron (~2 years ago), IIRC because some spawned process was incompatible with symlinks. It's the only time it caused me trouble though, it's indeed much faster than npm. I'd love to use it at work too.
  - zkochan 5y ago
    In any cases where pnpm doesn't work, you may set the node-linker=hoisted option and it will work: https://pnpm.io/npmrc#node-linker https://pnpm.io/npmrc#node-linker With node-linker=hoisted, pnpm creates a traditional hoisted (aka flat) node_modules, without using symlinks.
    - scambier 5y ago
      Good to know, thanks!
koolba 5y ago
Does this use symlinks to a single copy of each dependency?
- jessaustin 5y ago
  A single copy of each required version of each dependency.
  - koolba 5y ago
    So is that a yes? Or does it still pull a copy for each projects local node_modules?
    - jessaustin 5y ago
      As you can see at the URL in sibling, pnpm maintains a private area where everything is stored. Everything in a local (or "global", really) node_modules tree is a link to the private store.
- zkochan 5y ago
  Not symlinks. Hard links. Or copy-on-write on systems that support them.
  - jessaustin 5y ago
    [EDIT: (thanks ttybird2! b^)] We might need to change the following diagram? https://pnpm.io/motivation https://pnpm.io/motivation It appears to show several symlinks? Thanks for pnpm!
    - ttybird2 5y ago
      You are actually replying to the pnpm maintainer :p
leerob 5y ago
We recently started sponsoring pnpm[1] as well as adding zero-config support for deployments and caching. I think pnpm is an incredible tool. Excited to see it grow further. [1]: https://vercel.com/changelog/projects-using-pnpm-can-now-be-deployed-with-zero-configuration https://vercel.com/changelog/projects-using-pnpm-can-now-be-...
- zkochan 5y ago
  Thank you!
brunojppb 5y ago
I recently migrated a large project from yarn to pnpm and the speed difference is insane. Everything related to dependencies runs must faster during local development AND CI. T The only tricky thing is that we had some issues with some dependencies that could not work properly. But using the “—-shamefully-hoist” flag did the trick. Everything works.
- eyelidlessness 5y ago
  You can potentially solve this more narrowly with hoist-pattern[1] so you’ll still benefit from the stricter structure overall. Or possibly even overrides[2] depending on the issue. 1: https://pnpm.io/npmrc#hoist-pattern https://pnpm.io/npmrc#hoist-pattern 2: https://pnpm.io/package_json#pnpmoverrides https://pnpm.io/package_json#pnpmoverrides
andrew_ 5y ago
Love that this continues to be shared. One of the most exciting projects in the Node ecosystem.
ttybird2 5y ago
Please note that pnpm is currently blocking all traffic from Russia and Belarus https://twitter.com/pnpmjs/status/1498306992577957890 https://twitter.com/pnpmjs/status/1498306992577957890
- duxup 5y ago
  "We will unblock it when you stop the war and de-occupy all the Ukrainian territory." There's what, maybe a handful of people who can make that happen?
  - hbn 5y ago
    I'm sure Putin is reeling that he can't use an obscure package manager for his web projects
    - hirako2000 5y ago
      the idea is to anger the citizens so that they then turn their anger against Putin. it doesn't matter that Putin is not directly impacted. I fundamentally disagree with the blockade. this moves makes this project incompatible with open source licensing. if anyone is interested in this sub thread, it diverge from the discussion around the tool itself, but note that all the project standing for Ukraine in this manner are: - breaking one of the fundamental principle of open source - pretty risky to use. what if your locality also becomes block-listed. - demonstrates a poor judgment from their authors, who more often than not are reluctant to debate/reconsider their position. Reactions to extreme circumstances can understandably be emotional, it doesn't imply that logical criticisms are necessarily insensitive.
  - afavour 5y ago
    Russian TV is wall to wall coverage about their highly successful campaign to eradicate Ukrainian Nazis. An action like this might at least be a small hint to everyday Russians that things aren’t as they appear.
    - leshenka 5y ago
toastercat 5y ago
pnpm is great. I love `pnpm store prune`. Also it's great for monorepos, check out https://github.com/panoply/mithril-demo https://github.com/panoply/mithril-demo for an example.
rektide 5y ago
we're part way through switching all our monorepos from lerna to pnpm & simply could not be happier & more excited. lerna has soo many issues, is slow & cumbersome. #2142, no way to update dependecies in monorepo subprojects? how is there a monorepo tool where projects cant update their deps? everything on pnpm's built-in monorepo support just works, nice & easy & fast.
eyelidlessness 5y ago
I’m using PNPM in a side project to try it out. Overall, I like it. Yes, it does feel faster than Yarn (1, I gave up on Yarn 2 pretty quickly when it wouldn’t let me install a TypeScript beta even with PnP disabled). I’ve found its commands a little fussier to use than Yarn however. If I were to pick for a new project today, I’d probably go back to Yarn. But I’m glad there’s PNPM as an option.
steven-xu 5y ago
Summarizing the 3 major JS package management approaches: * Classic node_modules: Dependencies of dependencies that can't be satisfied by a shared hoisted version are nested as true copies (OSes may apply copy-on-write semantics on top of this, but from FS perspective, these are real files). Uses standard Node.js node_modules resolution [1]. * pnpm: ~1 real copy of each dependency version, and packages use symlinks in node_modules to point to their deps. Also uses standard resolution. Requires some compatibility work for packages that wrongly refer to transitive dependencies or to peer dependencies. * pnp[2]: 1 real copy of each dependency version, but it's a zip file with Node.js and related ecosystem packages patched to read from zips and traverse dependency edges using a sort of import map. In addition to the compatibility work required from pnpm, this further requires compatibility work around the zip "filesystem" indirection. In our real-world codebase, where we've done a modest but not exhaustive amount of package deduplication, pnpm confers around a 30% disk utilization savings, and pnp around a 80% savings. Interestingly, the innovations on top of classic node_modules are so compelling that the package managers that originally implemented pnpm and pnp (pnpm and Yarn, respectively) have implemented each others' linking strategies as optional configs [3][4]. If MacOS had better FUSE ergonomics, I'd be counting down the days for another linking strategy based on that too. [1] - https://nodejs.org/api/modules.html#loading-from-node_modules-folders https://nodejs.org/api/modules.html#loading-from-node_module... [2] - https://yarnpkg.com/features/pnp https://yarnpkg.com/features/pnp [3] - https://github.com/pnpm/pnpm/issues/2902 https://github.com/pnpm/pnpm/issues/2902 [4] - https://github.com/yarnpkg/berry/pull/3338 https://github.com/yarnpkg/berry/pull/3338
- raydiatian 5y ago
  That’s kind of incredible that yarn pnp out performs pnpm. If that’s generally true across most projects then I’m really glad that turborepo decided to use it for project subslicing.
  - jbverschoor 5y ago
    Problames long solved before, but problems that don't matter to the javascript crowd.. I think they actually love that things take so long. It makes them thing they're doing important work.. "We're compiling and initting"
forty 5y ago
If you have a pnpm monorepo, you need to know about https://pnpm.io/filtering#--filter-since https://pnpm.io/filtering#--filter-since which allows to run your test/lint/etc on only the packages that have been impacted by changes from master.
- mwid 5y ago
  I didn't know about this, thanks for posting! We have been using a custom Bash script to retrieve the list of sub-project directories that had been changed since the last commit by recursively scanning its sub-paths and running Git commands to find the last update on each, but this looks much, much better.
rplnt 5y ago
Slightly unrelated, but it always amazes me how big the dependency tree in js world can get. Hundreds of megabytes for SPAs. I don't understand js development or how anyone can live with that, but since I do need to edit it here and there I stumbled upon this handy tool. https://github.com/voidcosmos/npkill https://github.com/voidcosmos/npkill
- zkochan 5y ago
  It is unrelated because you don't have this issue with pnpm. pnpm uses a central content-addressable store and each unique file is written only once on a disk. It doesn't matter in how many projects you install the same dependency.
  - topaz0 5y ago
    I think GP was amazed by the sheer number of dependencies, not by the how many of them are duplicated. It also amazes me. People talking above about having 10s of GB of dependencies. Crazy to me.
    - crispyalmond 5y ago
      This amazes me as well. At work we have a simple website which has 1 GB of node_modules and the actual source code is well under a megabyte.
      - bavell 5y ago
        I'm very curious to know how in the world a node_modules directory can grow above 1GB! Which dependencies are the worst offenders? Is it dominated by a few huge deps?
rauttis 5y ago
I recently migrated a fairly large monorepo (20+ packages) that used Lerna and npm to pnpm, and the improvement in developer experience was pretty massive. Dependency install times went down by a huge amount, and all the strange issues we had with lerna and npm sometimes erroring out, requiring us to remove all node_modules folders and re-install everything, are just gone. We even noticed that the size of some of our production bundles went down. Before some dependencies that were used in multiple packages were being duplicated in our webpack bundles needlessly, but the way pnpm symlinks dependencies instead of duplicating them fixed that as well. The non-flat node_modules structure did break some things as well, since in some places we had imports pointing to packages that were transitive dependencies and not defined in package.json. I see this as a positive though, since all those cases were just bugs waiting to happen.
- krikou 5y ago
  I experienced the same. The overall dev experience / responsiveness of pakage management makes it very unlikely I would like to go back to npm ever.
- epolanski 5y ago
  Why not use rush as well?
  - girvo 5y ago
    Probably because Lerna is already working for them. We're just using straight Yarn workspaces ourselves. Rush is great, but for our project it's overkill (lerna was as well). Though we're using git submodules + yarn workspaces right now. The submodules will likely go away eventually.
xiphias2 5y ago
As a person who uses npm just for some hobby coding projects, it's quite frustrating that there are new partly incompatible package managers for the javascript ecosystem: npm, pnpm, yarn, yarn 2. Some packages need one, some another, so I tried to switch to yarn (or yarn 2) for a package that I wanted to try out, but then other packages stopped working. If there are clearly better algorithms, why not refactor npm and add them in experimental flags to npm and then setting them to default as they mature (with safe switching from one data structure or another)?
- Cu3PO42 5y ago
  For what it's worth Yarn 3 implements essentially all modes. It can do standard node_modules, its own Plug'n'Play, as well as pnpm-style hardlinking to a global cache. Edit: I just learned from another comment that PNPM also supports Plug'N'Play :) Thanks steven-xu!
  - zkochan 5y ago
    pnpm as well supports all three modes. But I think it is better to use Yarn for the PnP mode and pnpm for the symlinked mode. Here is a feature comparison: https://pnpm.io/feature-comparison https://pnpm.io/feature-comparison
  - xiphias2 5y ago
    So it should just be backported to npm to show that the authors are serious about backwards compatibility.
    - zkochan 5y ago
      Both pnpm and Yarn are independent projects maintained by the community. I personally think that these are better projects than npm CLI because they can make their own decisions. Not decisions dictated by business needs of a company. I was OK to merge pnpm into npm in the past. They have never suggested me this opportunity. Instead, they decided to re-implement pnpm's algorithm into npm and call it "isolated mode".
AtlasBarfed 5y ago
"Fast": compared to what? What is "fast"? "Disk space efficient": again, compared to what? Javascript: ...oh
zkochan 5y ago
pnpm is really good for monorepos and there are many big open source projects that use pnpm: https://pnpm.io/workspaces#usage-examples https://pnpm.io/workspaces#usage-examples
Chyzwar 5y ago
We migrated from pnpm to yarn3 with node_modules linker. pnpm focus on wrong things, speed, disk efficiently is less important than stable, reproducible, and developer experience of yarn3.
- physicsguy 5y ago
  We did exactly the same, tried to uneject from CRA recently and just could not get it to work in PNPM, switched to Yarn and it just worked.
tekkk 5y ago
I use pnpm only for monorepos for which I find it works quite well. Although at times there have been issues, mainly with symlinks getting messed up or linking to wrong dependencies. Well, mainly having multiple versions of same packages. But all in all I'm glad they are moving the JS/TS ecosystem forward and other managers are catching up to their innovations and likewise. Great monorepo support I feel is a big necessity for a manager and well-working symlinking to prevent the insane node_modules sizes.
kall 5y ago
I'm pretty immune to most JS ecosystem churn, but on package managers I'm feeling it. All I want is a package manager that is correct, causes very few day to day issues and is going to work the same way for many years. Yet everyone seems to be optimizing disk usage and speed without even getting the basics (make it work, make it right) fully covered. I don't understand why people are optimizing for disk space at all tbh. Like, have you ever edited a video project, used docker, installed Xcode? I cannot imagine what you must be doing for all node_modules combined to take up more than maybe 100 GB on disk. pnpm seems to be the lightest of the bunch, which is nice but why even mess with symlinks and confuse other tools? Just put all the files in there, nest them and duplicate them and everything. I'll happily live with my 10 GB node_modules folder that never breaks and sometimes gives me a nice coffee break. Possibly I'm actually just salty that Metro doesn't support symlinks and would otherwise be on the pnpm love-train.
- ramesh31 5y ago
  NPM just has too much institutional inertia to avoid. The moment you make the decision to use something else, you are simply trading one set of warts for another. I can't even tell you how many projects I have seen waste countless hours of dev time on Yarn/NPM discrepancies. If you are working on anything with more than two people, you really need to just use the standard tooling that everyone is familiar with and that the entire ecosystem is based around. Anything else is yak shaving.
  - aabbcc1241 5y ago
    I have custom bash scripts named npm and yarn, they invoke pnpm for installing and uninstalling packages, and fallback on other commands (e.g. audit). This way work well with other tools (e.g. I can force create-react-app to install packages with pnpm)
    - brainbag 5y ago
      There's an npm package "narn" that uses commands akin to yarn's but will automatically use whatever package manager the current folder(/parent) is using. I almost never type npm or yarn or pnpm, just narn everywhere. Really handy.
jimmy2020 5y ago
I've migrated from yarn to pnpm two days ago and I can tell the difference when I first hit install. I am working on a workspace so I have multiple packages with nested dependencies. For this case, I thought yarn (the classic version) is the ideal solution that was until I discovered pnpm. Thanks to its non-flat algorithm the packages have clean dependencies. Previously in yarn if you install `foo` in any package you can reuse it later in the workspace even if it's not listed in the package dependency. With pnpm it's not the case, which means a clean dependency tree and serving the purpose of what's workspace is meant for. If you want to share the dependency you can install it in the root which makes sense to me. Another big advantage is the recursive/parallel command something I couldn't do without Lerna. And it's fast. install once and it's there in the disk, so if you manage multiple projects, dependency installation is not something you wait for it's just there.
sam_goody 5y ago
with its linking strategy, pnpm allows for multiple versions of the same package. I wish there was something like that in PHP's Composer, where I have repeatedly hit situations where different packages have a dependency that was pinned to a different version (such as one package using Guzzle 6 and another Guzzle 7), and therefore my composer.json was un-buildable. (I have less experience with NPM so I don't know if they have a different solution for this.)
- tored 5y ago
  Problem is that it can’t be done in PHP without on-the-fly rewriting of source files. In JavaScript the caller decide how a module should be used by importing it to symbol, thus different versions of the same library can exist simultaneously. In PHP it is the callee that decides how to be imported because of namespaces. One and only one class with that exact name and namespace can exist at a given time, thus different versions of the same library can’t be loaded simultaneously. That is why modules are far superior over namespaces.
epolanski 5y ago
I see many mentioning using pnpm for monorepos, why not use rushjs by microsoft which is its natural companion for that use case?
pragmatic 5y ago
For us peasants stuck in angular land: https://github.com/pnpm/pnpm/issues/3410 https://github.com/pnpm/pnpm/issues/3410 Seems this might not work with angular (yet)?
- jmondi 5y ago
  As long as you are on Angular 13, and all of your dependencies are updated and use the Ivy view engine, you should be good to go. You'll get really weird, hard to debug issues if any of your dependencies are using the older view engine.
keyworks 5y ago
We switched our monorepo from yarn classic to pnpm and the install speeds have been night and day. Sometimes it feels like I ran the wrong command because its so fast. pnpm also helped us solve issues with packages in our monorepo importing dependencies declared in other packages because of node_modules hoisting behaviour with yarn. With yarn, all of your packages' direct and indirect dependencies can be resolved from any package in your monorepo. This makes it hard to isolate packages from each other. For example, this was causing issues when we wanted to generate docker images for some of our packages since we only wanted to copy specific directories into the image rather than the entire monorepo to avoid having to bring the entire monorepo to the image. pnpm also allows us to use features like git sparse-checkout because we can be confident there are no implicit dependencies between packages. pnpm makes this possible by only exposing in the node_modules direct dependencies. In monorepos that don't use pnpm, you can remove a dependency and your monorepo can break in unexpected places because other packages in the repo implicitly depend on this dependency. This is more common than you think because people tend to rely on IDE auto-completion when writing imports and IDEs just tend to read node_modules instead of package.json so its very easy to take on an implicit dependency when all directs/indirects are hoisted into a root node_modules. Yarn supposedly fixes this issue in their newer releases with Plug n Play but from from what I understand it basically monkey patches Node's require statements with its own behaviour. This just didn't feel good to me from a tooling compatibility perspective which is why I went with pnpm and I'm glad I did. I'm honestly puzzled at Yarn's popularity and why pnpm isn't as popular as it should be.
mrozbarry 5y ago
When I tried pnpm last, I was on a react native project (react 14 maybe), and something about the symlinks from pnpm caused me to not be able to build the project (and I went back to yarn). Anyone know if this is still an issue?
parentheses 5y ago
The way that key JS ecosystem programs get rewritten tells me nothing is made well. Most ecosystems don’t have this much constant change.

206 points by modinfo 5y ago

pnpm has been my goto JS monorepo package manager + script runner for a couple of years now. IMO it has almost zero downsides and huge upsides. I'd be surprised if npm doesn't end up adopting the pnpm node_modules directory hierarchy in the next 5 years or so.

no_wizard 5y ago
I believe the RFC for just that has landed: https://github.com/npm/rfcs/blob/main/accepted/0042-isolated-mode.md https://github.com/npm/rfcs/blob/main/accepted/0042-isolated... Yarn has a PNPM mode now too: https://dev.to/arcanis/yarn-31-corepack-esm-pnpm-optional-packages--3hak https://dev.to/arcanis/yarn-31-corepack-esm-pnpm-optional-pa...

synergy20 5y ago

This is my default one, _much_ faster and disk space reduced dramatically when I have lots of node_modules in use. Can't recommend enough.

eatonphil 5y ago
Were you coming from yarn or npm? I guess their benchmarks cover both [0]. But I'm also curious about independent figures. [0] https://pnpm.io/benchmarks https://pnpm.io/benchmarks
- synergy20 5y ago
  used to use npm, less often with yarn, now it's all pnpm
  - eatonphil 5y ago
    Gotcha. Yeah yarn has always been way faster to me than npm. And indeed in these benchmarks pnpm is much faster than npm and only slightly faster than yarn.

CSDude 5y ago

It works very fast in CI, its cache is smaller and it builds node_modules much faster. I don't feel comfortable caching node_modules folder itself, because I got side effects even causing incidents before, not sure if its supposed to. Speed difference is 15s vs 35s for our use case which is pretty significant.

hu3 5y ago

Is there any gotcha if I switch from npm to pnpm? And can I use pnpm when working in a team where other devs use npm?

jakubmazanec 5y ago
Pnpm doesn't auto install peer dependencies, which is annoying and forces you to unnecessary add them to package.json. Npm@7 (and newer) with auto install of peer deps is much easier to use in this regard.
- ricardobeat 5y ago
  Isn't that exactly how it's supposed to be? If you're installing a package that has peer dependencies, you should already depend on them. Otherwise they are not "peer dependencies" anymore, just a normal dependency with a version range. The whole peer dependencies story is another clusterfuck. Everyone simply ignored the invalid peer dependency warnings, and now npm itself will just install all of them in 'whatever works' version. To get real peer dependency resolution you need a --strict-peer-deps flag. Not exactly a "feature" in my book.
  - Wronnay 5y ago
    Yeah - this peer dependency thing always makes issues, when I try to upgrade a big Ionic project... Peer dependencies added more problems than they solved in my eyes... Most peer dependencies warnings come because some maintainer forgot to update the package.json and not really because two packages don't work with each other...
    - eyelidlessness 5y ago
      The point of peer dependencies is to allow users more flexibility in choosing the specific version of the dependency. It’s not intended to specify any kind of conflict between the packages, it’s effectively BYO sub-dependency.

yewenjie 5y ago

pnpm is awesome except when it does not work - which happens very rarely but it is a nightmare to debug. For all the other times, it is way faster and lightweight than npm.

scambier 5y ago
I was unable to use pnpm with a project that used Electron (~2 years ago), IIRC because some spawned process was incompatible with symlinks. It's the only time it caused me trouble though, it's indeed much faster than npm. I'd love to use it at work too.
- zkochan 5y ago
  In any cases where pnpm doesn't work, you may set the node-linker=hoisted option and it will work: https://pnpm.io/npmrc#node-linker https://pnpm.io/npmrc#node-linker With node-linker=hoisted, pnpm creates a traditional hoisted (aka flat) node_modules, without using symlinks.
  - scambier 5y ago
    Good to know, thanks!

koolba 5y ago

Does this use symlinks to a single copy of each dependency?

jessaustin 5y ago
A single copy of each required version of each dependency.
- koolba 5y ago
  So is that a yes? Or does it still pull a copy for each projects local node_modules?
  - jessaustin 5y ago
    As you can see at the URL in sibling, pnpm maintains a private area where everything is stored. Everything in a local (or "global", really) node_modules tree is a link to the private store.
zkochan 5y ago
Not symlinks. Hard links. Or copy-on-write on systems that support them.
- jessaustin 5y ago
  [EDIT: (thanks ttybird2! b^)] We might need to change the following diagram? https://pnpm.io/motivation https://pnpm.io/motivation It appears to show several symlinks? Thanks for pnpm!
  - ttybird2 5y ago
    You are actually replying to the pnpm maintainer :p

leerob 5y ago

We recently started sponsoring pnpm[1] as well as adding zero-config support for deployments and caching. I think pnpm is an incredible tool. Excited to see it grow further. [1]: https://vercel.com/changelog/projects-using-pnpm-can-now-be-deployed-with-zero-configuration https://vercel.com/changelog/projects-using-pnpm-can-now-be-...

zkochan 5y ago
Thank you!

brunojppb 5y ago

I recently migrated a large project from yarn to pnpm and the speed difference is insane. Everything related to dependencies runs must faster during local development AND CI. T The only tricky thing is that we had some issues with some dependencies that could not work properly. But using the “—-shamefully-hoist” flag did the trick. Everything works.

eyelidlessness 5y ago
You can potentially solve this more narrowly with hoist-pattern[1] so you’ll still benefit from the stricter structure overall. Or possibly even overrides[2] depending on the issue. 1: https://pnpm.io/npmrc#hoist-pattern https://pnpm.io/npmrc#hoist-pattern 2: https://pnpm.io/package_json#pnpmoverrides https://pnpm.io/package_json#pnpmoverrides

andrew_ 5y ago

Love that this continues to be shared. One of the most exciting projects in the Node ecosystem.

ttybird2 5y ago

Please note that pnpm is currently blocking all traffic from Russia and Belarus https://twitter.com/pnpmjs/status/1498306992577957890 https://twitter.com/pnpmjs/status/1498306992577957890

duxup 5y ago
"We will unblock it when you stop the war and de-occupy all the Ukrainian territory." There's what, maybe a handful of people who can make that happen?
- hbn 5y ago
  I'm sure Putin is reeling that he can't use an obscure package manager for his web projects
  - hirako2000 5y ago
    the idea is to anger the citizens so that they then turn their anger against Putin. it doesn't matter that Putin is not directly impacted. I fundamentally disagree with the blockade. this moves makes this project incompatible with open source licensing. if anyone is interested in this sub thread, it diverge from the discussion around the tool itself, but note that all the project standing for Ukraine in this manner are: - breaking one of the fundamental principle of open source - pretty risky to use. what if your locality also becomes block-listed. - demonstrates a poor judgment from their authors, who more often than not are reluctant to debate/reconsider their position. Reactions to extreme circumstances can understandably be emotional, it doesn't imply that logical criticisms are necessarily insensitive.
- afavour 5y ago
  Russian TV is wall to wall coverage about their highly successful campaign to eradicate Ukrainian Nazis. An action like this might at least be a small hint to everyday Russians that things aren’t as they appear.
  - leshenka 5y ago

toastercat 5y ago

pnpm is great. I love `pnpm store prune`. Also it's great for monorepos, check out https://github.com/panoply/mithril-demo https://github.com/panoply/mithril-demo for an example.

rektide 5y ago

we're part way through switching all our monorepos from lerna to pnpm & simply could not be happier & more excited. lerna has soo many issues, is slow & cumbersome. #2142, no way to update dependecies in monorepo subprojects? how is there a monorepo tool where projects cant update their deps? everything on pnpm's built-in monorepo support just works, nice & easy & fast.

eyelidlessness 5y ago

I’m using PNPM in a side project to try it out. Overall, I like it. Yes, it does feel faster than Yarn (1, I gave up on Yarn 2 pretty quickly when it wouldn’t let me install a TypeScript beta even with PnP disabled). I’ve found its commands a little fussier to use than Yarn however. If I were to pick for a new project today, I’d probably go back to Yarn. But I’m glad there’s PNPM as an option.

steven-xu 5y ago

Summarizing the 3 major JS package management approaches: * Classic node_modules: Dependencies of dependencies that can't be satisfied by a shared hoisted version are nested as true copies (OSes may apply copy-on-write semantics on top of this, but from FS perspective, these are real files). Uses standard Node.js node_modules resolution [1]. * pnpm: ~1 real copy of each dependency version, and packages use symlinks in node_modules to point to their deps. Also uses standard resolution. Requires some compatibility work for packages that wrongly refer to transitive dependencies or to peer dependencies. * pnp[2]: 1 real copy of each dependency version, but it's a zip file with Node.js and related ecosystem packages patched to read from zips and traverse dependency edges using a sort of import map. In addition to the compatibility work required from pnpm, this further requires compatibility work around the zip "filesystem" indirection. In our real-world codebase, where we've done a modest but not exhaustive amount of package deduplication, pnpm confers around a 30% disk utilization savings, and pnp around a 80% savings. Interestingly, the innovations on top of classic node_modules are so compelling that the package managers that originally implemented pnpm and pnp (pnpm and Yarn, respectively) have implemented each others' linking strategies as optional configs [3][4]. If MacOS had better FUSE ergonomics, I'd be counting down the days for another linking strategy based on that too. [1] - https://nodejs.org/api/modules.html#loading-from-node_modules-folders https://nodejs.org/api/modules.html#loading-from-node_module... [2] - https://yarnpkg.com/features/pnp https://yarnpkg.com/features/pnp [3] - https://github.com/pnpm/pnpm/issues/2902 https://github.com/pnpm/pnpm/issues/2902 [4] - https://github.com/yarnpkg/berry/pull/3338 https://github.com/yarnpkg/berry/pull/3338

raydiatian 5y ago
That’s kind of incredible that yarn pnp out performs pnpm. If that’s generally true across most projects then I’m really glad that turborepo decided to use it for project subslicing.
- jbverschoor 5y ago
  Problames long solved before, but problems that don't matter to the javascript crowd.. I think they actually love that things take so long. It makes them thing they're doing important work.. "We're compiling and initting"

forty 5y ago

If you have a pnpm monorepo, you need to know about https://pnpm.io/filtering#--filter-since https://pnpm.io/filtering#--filter-since which allows to run your test/lint/etc on only the packages that have been impacted by changes from master.

mwid 5y ago
I didn't know about this, thanks for posting! We have been using a custom Bash script to retrieve the list of sub-project directories that had been changed since the last commit by recursively scanning its sub-paths and running Git commands to find the last update on each, but this looks much, much better.

rplnt 5y ago

Slightly unrelated, but it always amazes me how big the dependency tree in js world can get. Hundreds of megabytes for SPAs. I don't understand js development or how anyone can live with that, but since I do need to edit it here and there I stumbled upon this handy tool. https://github.com/voidcosmos/npkill https://github.com/voidcosmos/npkill

zkochan 5y ago
It is unrelated because you don't have this issue with pnpm. pnpm uses a central content-addressable store and each unique file is written only once on a disk. It doesn't matter in how many projects you install the same dependency.
- topaz0 5y ago
  I think GP was amazed by the sheer number of dependencies, not by the how many of them are duplicated. It also amazes me. People talking above about having 10s of GB of dependencies. Crazy to me.
  - crispyalmond 5y ago
    This amazes me as well. At work we have a simple website which has 1 GB of node_modules and the actual source code is well under a megabyte.
    - bavell 5y ago
      I'm very curious to know how in the world a node_modules directory can grow above 1GB! Which dependencies are the worst offenders? Is it dominated by a few huge deps?

rauttis 5y ago

I recently migrated a fairly large monorepo (20+ packages) that used Lerna and npm to pnpm, and the improvement in developer experience was pretty massive. Dependency install times went down by a huge amount, and all the strange issues we had with lerna and npm sometimes erroring out, requiring us to remove all node_modules folders and re-install everything, are just gone. We even noticed that the size of some of our production bundles went down. Before some dependencies that were used in multiple packages were being duplicated in our webpack bundles needlessly, but the way pnpm symlinks dependencies instead of duplicating them fixed that as well. The non-flat node_modules structure did break some things as well, since in some places we had imports pointing to packages that were transitive dependencies and not defined in package.json. I see this as a positive though, since all those cases were just bugs waiting to happen.

krikou 5y ago
I experienced the same. The overall dev experience / responsiveness of pakage management makes it very unlikely I would like to go back to npm ever.
epolanski 5y ago
Why not use rush as well?
- girvo 5y ago
  Probably because Lerna is already working for them. We're just using straight Yarn workspaces ourselves. Rush is great, but for our project it's overkill (lerna was as well). Though we're using git submodules + yarn workspaces right now. The submodules will likely go away eventually.

xiphias2 5y ago

As a person who uses npm just for some hobby coding projects, it's quite frustrating that there are new partly incompatible package managers for the javascript ecosystem: npm, pnpm, yarn, yarn 2. Some packages need one, some another, so I tried to switch to yarn (or yarn 2) for a package that I wanted to try out, but then other packages stopped working. If there are clearly better algorithms, why not refactor npm and add them in experimental flags to npm and then setting them to default as they mature (with safe switching from one data structure or another)?

Cu3PO42 5y ago
For what it's worth Yarn 3 implements essentially all modes. It can do standard node_modules, its own Plug'n'Play, as well as pnpm-style hardlinking to a global cache. Edit: I just learned from another comment that PNPM also supports Plug'N'Play :) Thanks steven-xu!
- zkochan 5y ago
  pnpm as well supports all three modes. But I think it is better to use Yarn for the PnP mode and pnpm for the symlinked mode. Here is a feature comparison: https://pnpm.io/feature-comparison https://pnpm.io/feature-comparison
- xiphias2 5y ago
  So it should just be backported to npm to show that the authors are serious about backwards compatibility.
  - zkochan 5y ago
    Both pnpm and Yarn are independent projects maintained by the community. I personally think that these are better projects than npm CLI because they can make their own decisions. Not decisions dictated by business needs of a company. I was OK to merge pnpm into npm in the past. They have never suggested me this opportunity. Instead, they decided to re-implement pnpm's algorithm into npm and call it "isolated mode".

AtlasBarfed 5y ago

"Fast": compared to what? What is "fast"? "Disk space efficient": again, compared to what? Javascript: ...oh

zkochan 5y ago

pnpm is really good for monorepos and there are many big open source projects that use pnpm: https://pnpm.io/workspaces#usage-examples https://pnpm.io/workspaces#usage-examples

Chyzwar 5y ago

We migrated from pnpm to yarn3 with node_modules linker. pnpm focus on wrong things, speed, disk efficiently is less important than stable, reproducible, and developer experience of yarn3.

physicsguy 5y ago
We did exactly the same, tried to uneject from CRA recently and just could not get it to work in PNPM, switched to Yarn and it just worked.

tekkk 5y ago

I use pnpm only for monorepos for which I find it works quite well. Although at times there have been issues, mainly with symlinks getting messed up or linking to wrong dependencies. Well, mainly having multiple versions of same packages. But all in all I'm glad they are moving the JS/TS ecosystem forward and other managers are catching up to their innovations and likewise. Great monorepo support I feel is a big necessity for a manager and well-working symlinking to prevent the insane node_modules sizes.

kall 5y ago

I'm pretty immune to most JS ecosystem churn, but on package managers I'm feeling it. All I want is a package manager that is correct, causes very few day to day issues and is going to work the same way for many years. Yet everyone seems to be optimizing disk usage and speed without even getting the basics (make it work, make it right) fully covered. I don't understand why people are optimizing for disk space at all tbh. Like, have you ever edited a video project, used docker, installed Xcode? I cannot imagine what you must be doing for all node_modules combined to take up more than maybe 100 GB on disk. pnpm seems to be the lightest of the bunch, which is nice but why even mess with symlinks and confuse other tools? Just put all the files in there, nest them and duplicate them and everything. I'll happily live with my 10 GB node_modules folder that never breaks and sometimes gives me a nice coffee break. Possibly I'm actually just salty that Metro doesn't support symlinks and would otherwise be on the pnpm love-train.

ramesh31 5y ago
NPM just has too much institutional inertia to avoid. The moment you make the decision to use something else, you are simply trading one set of warts for another. I can't even tell you how many projects I have seen waste countless hours of dev time on Yarn/NPM discrepancies. If you are working on anything with more than two people, you really need to just use the standard tooling that everyone is familiar with and that the entire ecosystem is based around. Anything else is yak shaving.
- aabbcc1241 5y ago
  I have custom bash scripts named npm and yarn, they invoke pnpm for installing and uninstalling packages, and fallback on other commands (e.g. audit). This way work well with other tools (e.g. I can force create-react-app to install packages with pnpm)
  - brainbag 5y ago
    There's an npm package "narn" that uses commands akin to yarn's but will automatically use whatever package manager the current folder(/parent) is using. I almost never type npm or yarn or pnpm, just narn everywhere. Really handy.

jimmy2020 5y ago

I've migrated from yarn to pnpm two days ago and I can tell the difference when I first hit install. I am working on a workspace so I have multiple packages with nested dependencies. For this case, I thought yarn (the classic version) is the ideal solution that was until I discovered pnpm. Thanks to its non-flat algorithm the packages have clean dependencies. Previously in yarn if you install `foo` in any package you can reuse it later in the workspace even if it's not listed in the package dependency. With pnpm it's not the case, which means a clean dependency tree and serving the purpose of what's workspace is meant for. If you want to share the dependency you can install it in the root which makes sense to me. Another big advantage is the recursive/parallel command something I couldn't do without Lerna. And it's fast. install once and it's there in the disk, so if you manage multiple projects, dependency installation is not something you wait for it's just there.

sam_goody 5y ago

with its linking strategy, pnpm allows for multiple versions of the same package. I wish there was something like that in PHP's Composer, where I have repeatedly hit situations where different packages have a dependency that was pinned to a different version (such as one package using Guzzle 6 and another Guzzle 7), and therefore my composer.json was un-buildable. (I have less experience with NPM so I don't know if they have a different solution for this.)

tored 5y ago
Problem is that it can’t be done in PHP without on-the-fly rewriting of source files. In JavaScript the caller decide how a module should be used by importing it to symbol, thus different versions of the same library can exist simultaneously. In PHP it is the callee that decides how to be imported because of namespaces. One and only one class with that exact name and namespace can exist at a given time, thus different versions of the same library can’t be loaded simultaneously. That is why modules are far superior over namespaces.

epolanski 5y ago

I see many mentioning using pnpm for monorepos, why not use rushjs by microsoft which is its natural companion for that use case?

pragmatic 5y ago

For us peasants stuck in angular land: https://github.com/pnpm/pnpm/issues/3410 https://github.com/pnpm/pnpm/issues/3410 Seems this might not work with angular (yet)?

jmondi 5y ago
As long as you are on Angular 13, and all of your dependencies are updated and use the Ivy view engine, you should be good to go. You'll get really weird, hard to debug issues if any of your dependencies are using the older view engine.

keyworks 5y ago

We switched our monorepo from yarn classic to pnpm and the install speeds have been night and day. Sometimes it feels like I ran the wrong command because its so fast. pnpm also helped us solve issues with packages in our monorepo importing dependencies declared in other packages because of node_modules hoisting behaviour with yarn. With yarn, all of your packages' direct and indirect dependencies can be resolved from any package in your monorepo. This makes it hard to isolate packages from each other. For example, this was causing issues when we wanted to generate docker images for some of our packages since we only wanted to copy specific directories into the image rather than the entire monorepo to avoid having to bring the entire monorepo to the image. pnpm also allows us to use features like git sparse-checkout because we can be confident there are no implicit dependencies between packages. pnpm makes this possible by only exposing in the node_modules direct dependencies. In monorepos that don't use pnpm, you can remove a dependency and your monorepo can break in unexpected places because other packages in the repo implicitly depend on this dependency. This is more common than you think because people tend to rely on IDE auto-completion when writing imports and IDEs just tend to read node_modules instead of package.json so its very easy to take on an implicit dependency when all directs/indirects are hoisted into a root node_modules. Yarn supposedly fixes this issue in their newer releases with Plug n Play but from from what I understand it basically monkey patches Node's require statements with its own behaviour. This just didn't feel good to me from a tooling compatibility perspective which is why I went with pnpm and I'm glad I did. I'm honestly puzzled at Yarn's popularity and why pnpm isn't as popular as it should be.

mrozbarry 5y ago

When I tried pnpm last, I was on a react native project (react 14 maybe), and something about the symlinks from pnpm caused me to not be able to build the project (and I went back to yarn). Anyone know if this is still an issue?

parentheses 5y ago

The way that key JS ecosystem programs get rewritten tells me nothing is made well. Most ecosystems don’t have this much constant change.