5 ms·

As a complete nix noob, will this help with caching node dependencies? We have a few projects that take over 20mins for a `yarn install && yarn build`. I’ve rea

As a complete nix noob, will this help with caching node dependencies? We have a few projects that take over 20mins for a `yarn install && yarn build`. I’ve read setting up Nix for node isn’t that straightforward, but that was a couple of years back. Has anything changed with respect to node projects?

biggestlou 3y ago
You may find something like node2nix helpful (https://github.com/svanderburg/node2nix https://github.com/svanderburg/node2nix). This converts your package.json into a Nix expression that can then be cached. You're right that it does require some setup and a bit of Nix knowledge but could yield significant benefits and take a good chunk out of that 20 minutes. Another option might be to use pnpm instead of Yarn and cache your pnpm dependencies. pnpm actually works a bit like Nix in that it creates a pnpm-lock.yaml file with content-based hashes for the full package.json dependency tree. This enables it to quickly determine which parts of the dependency tree it needs to build and which are already available.
- soraminazuki 3y ago
  Nowadays, there's buildNpmPackage [1]. It's included in Nixpkgs, actively maintained, and easier to work with IMO. [1]: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/javascript.section.md#buildnpmpackage-javascript-buildnpmpackage https://github.com/NixOS/nixpkgs/blob/master/doc/languages-f...
duped 3y ago
In order to support resolving dependencies to multiple versions (eg A and B depend on C but at incompatible versions), what npm does is drop the version override for C inside the nested node_modules directory (node_modules/A/node_modules/C, for example). That means that node_modules (as created by a package-lock.json) can't really be cached or be built from caches since it depends on the particular version solution found by npm for a particular project. So there's only so much that Nix can do. It can cache it about as well as using a naive caching scheme with actions/upload-artifact or similar (create a tarball of your node_modules and just cache it across runs, update when you need to). Basically node_modules is inherently large. If you want better performance for caching dependencies use a better programming language environment.
- crabmusket 3y ago
  Or fewer dependencies!
maxloh 3y ago
That is supported officially: - uses: actions/cache@v3 with: # npm cache files are stored in `~/.npm` on Linux/macOS path: ~/.npm key: ${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
wereHamster 3y ago
I have given up installing node dependencies via Nix. I use Nix to make nodejs available but then install node deps with npm/yarn/pnpm. Node packages sometimes pull additional files from the internet in a postinstall script, or do other funky stuff that's incompatible with Nix. So the idea that you can construct a pure derivation from a package-lock.json or yarn.lock file is a pipe dream.

biggestlou 3y ago

You may find something like node2nix helpful (https://github.com/svanderburg/node2nix https://github.com/svanderburg/node2nix). This converts your package.json into a Nix expression that can then be cached. You're right that it does require some setup and a bit of Nix knowledge but could yield significant benefits and take a good chunk out of that 20 minutes. Another option might be to use pnpm instead of Yarn and cache your pnpm dependencies. pnpm actually works a bit like Nix in that it creates a pnpm-lock.yaml file with content-based hashes for the full package.json dependency tree. This enables it to quickly determine which parts of the dependency tree it needs to build and which are already available.

soraminazuki 3y ago
Nowadays, there's buildNpmPackage [1]. It's included in Nixpkgs, actively maintained, and easier to work with IMO. [1]: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/javascript.section.md#buildnpmpackage-javascript-buildnpmpackage https://github.com/NixOS/nixpkgs/blob/master/doc/languages-f...

duped 3y ago

In order to support resolving dependencies to multiple versions (eg A and B depend on C but at incompatible versions), what npm does is drop the version override for C inside the nested node_modules directory (node_modules/A/node_modules/C, for example). That means that node_modules (as created by a package-lock.json) can't really be cached or be built from caches since it depends on the particular version solution found by npm for a particular project. So there's only so much that Nix can do. It can cache it about as well as using a naive caching scheme with actions/upload-artifact or similar (create a tarball of your node_modules and just cache it across runs, update when you need to). Basically node_modules is inherently large. If you want better performance for caching dependencies use a better programming language environment.

crabmusket 3y ago
Or fewer dependencies!

maxloh 3y ago

That is supported officially: - uses: actions/cache@v3 with: # npm cache files are stored in `~/.npm` on Linux/macOS path: ~/.npm key: ${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}

wereHamster 3y ago

I have given up installing node dependencies via Nix. I use Nix to make nodejs available but then install node deps with npm/yarn/pnpm. Node packages sometimes pull additional files from the internet in a postinstall script, or do other funky stuff that's incompatible with Nix. So the idea that you can construct a pure derivation from a package-lock.json or yarn.lock file is a pipe dream.