7 ms·

The Magic Nix Cache, a GitHub Action for speeding up your Nix workflows

195 points by biggestlou 3y ago

bketelsen 3y ago
this looks awesome, can't wait to try it out.
grhmc 3y ago
Graham Christensen here, cofounder of DetSys. Happy to answer any questions! The Magic Nix Cache has been a huge boon to us internally, and we're really excited to share it with the world today.
- zetalyrae 3y ago
  Does it work with nix-shell? I don't know how to use flakes yet.
  - grhmc 3y ago
    Great question. Yes. Anything that Nix builds during your workflow will get cached. Give it a try and let me know how it goes?
  - biggestlou 3y ago
    Yes! The Action isn't directly aware of what Nix itself is doing or which commands are being run; it's only aware of the Nix store. So whether you're using flakes or channels it works the same.
- mikepurvis 3y ago
  As someone who has led a ~2yr adoption of Nix in an enterprise scenario, I'm really glad for more projects that help with bridging the gap between Nix and containers/k8s. You can get fast startups if you're willing to define your containers upfront (dockerTools, nix2container) and/or adopt a dynamic container-server (nixery, flakehub). And you can get reasonably fast substitutions if your binary cache is on MinIO in the same cluster as the workers. But I feel like there's still room for a "magic" /nix/store that skips the copying and decompression stage altogether— something that works using standard nix invocations (like Magic Nix Cache), but presents itself as a Kubernetes Volume, so that in cases where a path already exists on-node, the existing files (in the cache pod) are simply mounted/served directly into whatever container ran a nix command. I don't feel like I really know enough about either k8s or nix to assess the practicality of such a thing, but the thought of lightning-fast substitutions for arbitrary Nix workflows is massively appealing.
  - grhmc 3y ago
    It is great to hear about your work in the space! Several years ago I looked at implementing a custom component for k8s which would exchange nix store paths instead of containers, substitute, and bind mount them in at run time. It was an interesting experiment, but was Quite Difficult to pull off for someone who wasn't already familiar with k8s. I've seen some projects similar to what you're describing though: the lightning-fast substitutions. It was incredible! They had the benefit of a fabulously fat network connection, though, and I'm not sure the experience translates very well. We will see!
    - mikepurvis 3y ago
      Thanks! The domain is robotics, and we gave a brief conference talk last fall about our experiences so far, if you're interested— although it's mostly directed at a Nix-unaware audience, you can get a glimpse of our workflow that's heavily oriented around hourly builds delivered as Nix flake tags: https://vimeo.com/767139940 https://vimeo.com/767139940 Definitely there's an obviousness to the concept of "magic" Nix stores in various spaces, and I know the tvix project seeks to realize some of this as well— reusing OCI tools to supply the build sandbox, leveraging existing container orchestrators for job management and queuing, all those goodies. So I'm excited to keep watching the space.
  - saghm 3y ago
    > But I feel like there's still room for a "magic" /nix/store that skips the copying and decompression stage altogether— something that works using standard nix invocations (like Magic Nix Cache), but presents itself as a Kubernetes Volume, so that in cases where a path already exists on-node, the existing files (in the cache pod) are simply mounted/served directly into whatever container ran a nix command. Mostly a digression since it's not Nix, but I've wondered a bit about this sort of thing when building AUR packages on Arch Linux. The very last step of the builder is to compress the package, which in the vast majority of my uses is followed immediately by installing it, which of course decompresses the package. I've wondered why there isn't some (non-default) option to say "I don't need to keep the package itself around; just install it as soon as its built". I'm sure for my specific use case there's a simpler solution, but I've always wondered if there's a hacky way to get around things more generally by making a tool that can mimic the expected compression API but then creates a "fake" compressed artifact that no-ops (or maybe puts in a valid header followed by non-compressed data) and then injects that implementation into the PATH. You'd be able to invoke it with something like `fakecompress --zstd makepkg -si`, and it would invoke `makepkg -si` with the no-op zstd implementation.
- mananaysiempre 3y ago
  Typo: “speed future up Nix workflow runs”.
miduil 3y ago
Nice, really hope I'll find some sweet way to cache similarish with GitLab-CI. Also kinda been thinking about how cool it'd be to run Kubernetes with Nix natively (so instead of a docker layer registry you have nix paths mounted together to overlayfs)
- grhmc 3y ago
  I think it should be pretty straightforward to make the Magic Nix Cache work on GitLab, too. They have a similar caching API. We'll take a look!
  - randomblast 3y ago
    Is the same true for Azure Pipelines? Given Actions forked from Pipelines I'd imagine it would be straightforward.
    - grhmc 3y ago
      I'm not sure... want to open a ticket? :)
  - ivanbakel 3y ago
    I would be thrilled to see this ported to GitLab CI. This is an incredibly (for lack of a better word) sexy tool on top of Nix, and would perfectly fit a use case we have for using Nix for caching build artifacts.
  - pxc 3y ago
    I would love to use this kind of thing with GitLab CI (and AzDO) at work!
  - tracnar 3y ago
    Nice. The GitLab caching API is basically saving and restoring a folder, so if you allow pointing to a local folder instead of the GitHub API you should be good!
- whateveracct 3y ago
deleted 3y ago
[deleted]
grhmc 3y ago
One project cut their CI time from 18m to 3m: https://github.com/awkward-squad/hasql-interpolate/actions https://github.com/awkward-squad/hasql-interpolate/actions. I wonder who will see the biggest cut! Note that when PRs merge to the default branch, their cache doesn't come with them. This is how GitHub Action's cache works, as a security measure. However: subsequent rebuilds will, and PRs off the default branch will too.
iso8859-1 3y ago
How does the Magic Nix Cache know which parts of my build are deterministic and which aren't? I suppose maybe it will only work if I split my build up into multiple steps such that Nix will know to skip those first steps. If Nix knows that, I suppose the Magic Nix Cache also knows?
- biggestlou 3y ago
  Could you possibly clarify what you mean by "skip those first steps?" Which steps are you referring to?
  - iso8859-1 3y ago
    Presumably, the Magic Nix Cache speeds up your build by caching the output of the "first steps" (really leaves in a tree of inputs). Otherwise, how would it work? So by steps, I am referring to the steps that your Nix derivation consists of.
    - biggestlou 3y ago
      When Nix "realises" a derivation, it realises the entire dependency tree (here, "realises" means either building or fetching, depending on whether a dependency is already in the Nix store). For every single derivation (dependency) in the tree, Nix first calculates what the store path for that dependency would be and then uses that to determine if it's already stored in the Nix store. So it would look at, say, a glibc dependency in a derivation and determine that the Nix store path would be /nix/store/7kn2mkg0g49lfflkdip7i39q3zsck4pc-glibc. If that's already in the Nix store then it doesn't need to build that. And it applies this logic throughout the entire dependency tree. In some cases, the entire dependency tree has already been built and written to the Nix store, in which Nix knows that it doesn't have to build anything. So Nix's caching logic doesn't apply only to the "first steps" of a Nix build (or realisation, to be more specific); it applies to all steps.
      - grhmc 3y ago
        Check out https://zero-to-nix.com/concepts/closures https://zero-to-nix.com/concepts/closures and https://zero-to-nix.com/concepts/realisation https://zero-to-nix.com/concepts/realisation, which I think add more color to this (and also is largely authored by biggestlou here.)
dlahoda 3y ago
i replaced cloud autoscaled docker runners with fixed set of machines with nix store cache pee machine. 10x imrovement.
- grhmc 3y ago
  Yep, that works great for cases where you trust all your contributors. We do the same for some projects internally, too. However, for public projects where you don't it gets a bit ... dicier. Glad to hear that's working for you!
  - dlahoda 3y ago
    we have public oss project. i guess we were not attacked yet. prs from forks run only main workflows, and these run nix which is kind of isolated enough. i guess one could attack with some infinite nix store bomb.
- esafak 3y ago
  Are you saying you replaced Docker with Nix? I am still trying to understand the use case for Nix. Can someone explain? Is it a complete replacement for Docker?
  - rgoulter 3y ago
    > Is it a complete replacement for Docker? Docker mainly does two things: there's Docker images as a way of sharing some container-image, and the container runtime to allow running those images for tasks or services in isolated/fresh ways. Docker images make it easy to distribute software which runs the same everywhere. Nix is a package manager which tackles that problem, but without using container images: Nix is for distributing software so it has the same behaviour everywhere. Nix users are often very enthusiastic about Nix because it also enables all sorts of neat developer experiences. e.g. Nix is great for setting up development environments. In terms of Nix-vs-Docker, Nix is also capable of building Docker/OCI images. So, you could use Nix instead of writing a Dockerfile.
    - 3y ago
Karupan 3y ago
As a complete nix noob, will this help with caching node dependencies? We have a few projects that take over 20mins for a `yarn install && yarn build`. I’ve read setting up Nix for node isn’t that straightforward, but that was a couple of years back. Has anything changed with respect to node projects?
- biggestlou 3y ago
  You may find something like node2nix helpful (https://github.com/svanderburg/node2nix https://github.com/svanderburg/node2nix). This converts your package.json into a Nix expression that can then be cached. You're right that it does require some setup and a bit of Nix knowledge but could yield significant benefits and take a good chunk out of that 20 minutes. Another option might be to use pnpm instead of Yarn and cache your pnpm dependencies. pnpm actually works a bit like Nix in that it creates a pnpm-lock.yaml file with content-based hashes for the full package.json dependency tree. This enables it to quickly determine which parts of the dependency tree it needs to build and which are already available.
  - soraminazuki 3y ago
    Nowadays, there's buildNpmPackage [1]. It's included in Nixpkgs, actively maintained, and easier to work with IMO. [1]: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/javascript.section.md#buildnpmpackage-javascript-buildnpmpackage https://github.com/NixOS/nixpkgs/blob/master/doc/languages-f...
- duped 3y ago
  In order to support resolving dependencies to multiple versions (eg A and B depend on C but at incompatible versions), what npm does is drop the version override for C inside the nested node_modules directory (node_modules/A/node_modules/C, for example). That means that node_modules (as created by a package-lock.json) can't really be cached or be built from caches since it depends on the particular version solution found by npm for a particular project. So there's only so much that Nix can do. It can cache it about as well as using a naive caching scheme with actions/upload-artifact or similar (create a tarball of your node_modules and just cache it across runs, update when you need to). Basically node_modules is inherently large. If you want better performance for caching dependencies use a better programming language environment.
SirensOfTitan 3y ago
Slightly off-topic, but how do folks around here create production packages in nix? I use nix for my dev shells and machine configuration, but haven't yet built production packages using it. More concretely, let's say you have a python backend that uses poetry. Do you just use `poetry install` in your derivation for python-deps? Do you use something like poetry2nix or node2nix and do all of your package management in nix?
- grhmc 3y ago
  "It depends." poetry2nix is pretty good! Generally, exporting to an OCI image, or targeting a production environment that can run Nix closures natively is the way we go about it. Many of our services use buildLayeredImage[0] and target Fly.io. 0: https://grahamc.com/blog/nix-and-layered-docker-images/ https://grahamc.com/blog/nix-and-layered-docker-images/
  - flurie 3y ago
    > targeting a production environment that can run Nix closures natively Do you have some examples for this? The ones that come to mind are things like bare metal/vms with nix, or perhaps disnix, but those are a pretty hard sell over more popular orchestration systems, and I’d like to have more alternatives.
- takeda 3y ago
  So I so far use it for dev environment and also for CI pipeline (the benefit is that Nix allows caching between runs, it helps with things like: - getting dev environment on CI to be identical to user dev - with minor changes the project is not rebuilt or or rebuilt minimally - the caching works across branches, so for example merging a feature branch to master, if nothing changes the build on master will be very quick I created something similar to nix-cache for gitlab, but I had to create a dedicated runner running NixOS. If I could use NixOS for deployment, at that point I would just point the same binary cache to the machine and use the same derivation to build the app. Because the app was already build by CI, it would just download the compiled version. No need for artifactory or similar. In that scenario (you using poetry) you probably would just use poetry2nix to generate the application. If the OS is not NixOS, but you still want to deploy via nix, then IMO this[2] looks interesting, basically it packages everything in self extracting archive. That you can extract and then run the app. Other alternatives are these bundlers[3], which includes building toArx (works in a way similar to the previous one but pretends everything is in a single file), RPM, DEB, docker (you would have more control over it if you would use the code directly instead of a bundler though) And the last option (probably the most obvious one) is that you can simply just use the tool to build the package. Since you're using poetry, then you can generate a wheel from it. [1] https://github.com/takeda/nix-cde/blob/master/contrib/gitlab-runner/nix-builder.yaml https://github.com/takeda/nix-cde/blob/master/contrib/gitlab... [2] https://github.com/Ninlives/relocatable.nix https://github.com/Ninlives/relocatable.nix [3] https://github.com/NixOS/bundlers/blob/master/flake.nix https://github.com/NixOS/bundlers/blob/master/flake.nix
sgammon 3y ago
this is awesome. i've yet to try Nix, but we are starting a remote cache service and we'd love to be compatible with Nix' caching mechanism we're at https://less.build https://less.build if you want to take a peek -- we will look at adding S3 support! :)
- sgammon 3y ago
  side note i am happy anytime build caching is on the front page of hckrnews for any reason
- a_t48 3y ago
  I'm all for faster builds...but how does your product differ from using something like https://ccache.dev/manual/4.7.html#_remote_storage_backends https://ccache.dev/manual/4.7.html#_remote_storage_backends ? (I ask specifically because you advertise being a drop in ccache replacement, or similar)
  - sgammon 3y ago
    we are drop-in compatible with ccache, in the sense that you can plug ccache right in (as an HTTP backend), and gain features like authorization policies, analytics, and global caching backed by Cloudflare. that's where we are beginning and then we're working to add spicier features on from there. but first and foremost, we want to serve the ccache/gradle cache crowd with a fantastic protocol-agnostic backend which "just works" and pays for itself in terms of time saved. and thanks for asking :) we are very new
    - a_t48 3y ago
      Ah, cool. Not my usecase, but neat all the same.
wereHamster 3y ago
Just this past week I thought about setting up custom github runners on NixOS machines so that Nix is pre-installed (doesn't need to be installed via a github action) and so that the Nix store can be shared between runs. Though I don't really want to manage the machines, so I might give this new github action a try…
- yakshaving_jgt 3y ago
  You might be interested in this then. https://github-runners.www-cachix-org.pages.dev/github-runners-for-nix-nixos https://github-runners.www-cachix-org.pages.dev/github-runne...

195 points by biggestlou 3y ago

this looks awesome, can't wait to try it out.

Graham Christensen here, cofounder of DetSys. Happy to answer any questions! The Magic Nix Cache has been a huge boon to us internally, and we're really excited to share it with the world today.

zetalyrae 3y ago
Does it work with nix-shell? I don't know how to use flakes yet.
- grhmc 3y ago
  Great question. Yes. Anything that Nix builds during your workflow will get cached. Give it a try and let me know how it goes?
- biggestlou 3y ago
  Yes! The Action isn't directly aware of what Nix itself is doing or which commands are being run; it's only aware of the Nix store. So whether you're using flakes or channels it works the same.
mikepurvis 3y ago
As someone who has led a ~2yr adoption of Nix in an enterprise scenario, I'm really glad for more projects that help with bridging the gap between Nix and containers/k8s. You can get fast startups if you're willing to define your containers upfront (dockerTools, nix2container) and/or adopt a dynamic container-server (nixery, flakehub). And you can get reasonably fast substitutions if your binary cache is on MinIO in the same cluster as the workers. But I feel like there's still room for a "magic" /nix/store that skips the copying and decompression stage altogether— something that works using standard nix invocations (like Magic Nix Cache), but presents itself as a Kubernetes Volume, so that in cases where a path already exists on-node, the existing files (in the cache pod) are simply mounted/served directly into whatever container ran a nix command. I don't feel like I really know enough about either k8s or nix to assess the practicality of such a thing, but the thought of lightning-fast substitutions for arbitrary Nix workflows is massively appealing.
- grhmc 3y ago
  It is great to hear about your work in the space! Several years ago I looked at implementing a custom component for k8s which would exchange nix store paths instead of containers, substitute, and bind mount them in at run time. It was an interesting experiment, but was Quite Difficult to pull off for someone who wasn't already familiar with k8s. I've seen some projects similar to what you're describing though: the lightning-fast substitutions. It was incredible! They had the benefit of a fabulously fat network connection, though, and I'm not sure the experience translates very well. We will see!
  - mikepurvis 3y ago
    Thanks! The domain is robotics, and we gave a brief conference talk last fall about our experiences so far, if you're interested— although it's mostly directed at a Nix-unaware audience, you can get a glimpse of our workflow that's heavily oriented around hourly builds delivered as Nix flake tags: https://vimeo.com/767139940 https://vimeo.com/767139940 Definitely there's an obviousness to the concept of "magic" Nix stores in various spaces, and I know the tvix project seeks to realize some of this as well— reusing OCI tools to supply the build sandbox, leveraging existing container orchestrators for job management and queuing, all those goodies. So I'm excited to keep watching the space.
- saghm 3y ago
  > But I feel like there's still room for a "magic" /nix/store that skips the copying and decompression stage altogether— something that works using standard nix invocations (like Magic Nix Cache), but presents itself as a Kubernetes Volume, so that in cases where a path already exists on-node, the existing files (in the cache pod) are simply mounted/served directly into whatever container ran a nix command. Mostly a digression since it's not Nix, but I've wondered a bit about this sort of thing when building AUR packages on Arch Linux. The very last step of the builder is to compress the package, which in the vast majority of my uses is followed immediately by installing it, which of course decompresses the package. I've wondered why there isn't some (non-default) option to say "I don't need to keep the package itself around; just install it as soon as its built". I'm sure for my specific use case there's a simpler solution, but I've always wondered if there's a hacky way to get around things more generally by making a tool that can mimic the expected compression API but then creates a "fake" compressed artifact that no-ops (or maybe puts in a valid header followed by non-compressed data) and then injects that implementation into the PATH. You'd be able to invoke it with something like `fakecompress --zstd makepkg -si`, and it would invoke `makepkg -si` with the no-op zstd implementation.
mananaysiempre 3y ago
Typo: “speed future up Nix workflow runs”.

miduil 3y ago

Nice, really hope I'll find some sweet way to cache similarish with GitLab-CI. Also kinda been thinking about how cool it'd be to run Kubernetes with Nix natively (so instead of a docker layer registry you have nix paths mounted together to overlayfs)

grhmc 3y ago
I think it should be pretty straightforward to make the Magic Nix Cache work on GitLab, too. They have a similar caching API. We'll take a look!
- randomblast 3y ago
  Is the same true for Azure Pipelines? Given Actions forked from Pipelines I'd imagine it would be straightforward.
  - grhmc 3y ago
    I'm not sure... want to open a ticket? :)
- ivanbakel 3y ago
  I would be thrilled to see this ported to GitLab CI. This is an incredibly (for lack of a better word) sexy tool on top of Nix, and would perfectly fit a use case we have for using Nix for caching build artifacts.
- pxc 3y ago
  I would love to use this kind of thing with GitLab CI (and AzDO) at work!
- tracnar 3y ago
  Nice. The GitLab caching API is basically saving and restoring a folder, so if you allow pointing to a local folder instead of the GitHub API you should be good!
whateveracct 3y ago

deleted 3y ago

[deleted]

grhmc 3y ago

One project cut their CI time from 18m to 3m: https://github.com/awkward-squad/hasql-interpolate/actions https://github.com/awkward-squad/hasql-interpolate/actions. I wonder who will see the biggest cut! Note that when PRs merge to the default branch, their cache doesn't come with them. This is how GitHub Action's cache works, as a security measure. However: subsequent rebuilds will, and PRs off the default branch will too.

iso8859-1 3y ago

How does the Magic Nix Cache know which parts of my build are deterministic and which aren't? I suppose maybe it will only work if I split my build up into multiple steps such that Nix will know to skip those first steps. If Nix knows that, I suppose the Magic Nix Cache also knows?

biggestlou 3y ago
Could you possibly clarify what you mean by "skip those first steps?" Which steps are you referring to?
- iso8859-1 3y ago
  Presumably, the Magic Nix Cache speeds up your build by caching the output of the "first steps" (really leaves in a tree of inputs). Otherwise, how would it work? So by steps, I am referring to the steps that your Nix derivation consists of.
  - biggestlou 3y ago
    When Nix "realises" a derivation, it realises the entire dependency tree (here, "realises" means either building or fetching, depending on whether a dependency is already in the Nix store). For every single derivation (dependency) in the tree, Nix first calculates what the store path for that dependency would be and then uses that to determine if it's already stored in the Nix store. So it would look at, say, a glibc dependency in a derivation and determine that the Nix store path would be /nix/store/7kn2mkg0g49lfflkdip7i39q3zsck4pc-glibc. If that's already in the Nix store then it doesn't need to build that. And it applies this logic throughout the entire dependency tree. In some cases, the entire dependency tree has already been built and written to the Nix store, in which Nix knows that it doesn't have to build anything. So Nix's caching logic doesn't apply only to the "first steps" of a Nix build (or realisation, to be more specific); it applies to all steps.
    - grhmc 3y ago
      Check out https://zero-to-nix.com/concepts/closures https://zero-to-nix.com/concepts/closures and https://zero-to-nix.com/concepts/realisation https://zero-to-nix.com/concepts/realisation, which I think add more color to this (and also is largely authored by biggestlou here.)

dlahoda 3y ago

i replaced cloud autoscaled docker runners with fixed set of machines with nix store cache pee machine. 10x imrovement.

grhmc 3y ago
Yep, that works great for cases where you trust all your contributors. We do the same for some projects internally, too. However, for public projects where you don't it gets a bit ... dicier. Glad to hear that's working for you!
- dlahoda 3y ago
  we have public oss project. i guess we were not attacked yet. prs from forks run only main workflows, and these run nix which is kind of isolated enough. i guess one could attack with some infinite nix store bomb.
esafak 3y ago
Are you saying you replaced Docker with Nix? I am still trying to understand the use case for Nix. Can someone explain? Is it a complete replacement for Docker?
- rgoulter 3y ago
  > Is it a complete replacement for Docker? Docker mainly does two things: there's Docker images as a way of sharing some container-image, and the container runtime to allow running those images for tasks or services in isolated/fresh ways. Docker images make it easy to distribute software which runs the same everywhere. Nix is a package manager which tackles that problem, but without using container images: Nix is for distributing software so it has the same behaviour everywhere. Nix users are often very enthusiastic about Nix because it also enables all sorts of neat developer experiences. e.g. Nix is great for setting up development environments. In terms of Nix-vs-Docker, Nix is also capable of building Docker/OCI images. So, you could use Nix instead of writing a Dockerfile.
  - 3y ago

Karupan 3y ago

As a complete nix noob, will this help with caching node dependencies? We have a few projects that take over 20mins for a `yarn install && yarn build`. I’ve read setting up Nix for node isn’t that straightforward, but that was a couple of years back. Has anything changed with respect to node projects?

biggestlou 3y ago
You may find something like node2nix helpful (https://github.com/svanderburg/node2nix https://github.com/svanderburg/node2nix). This converts your package.json into a Nix expression that can then be cached. You're right that it does require some setup and a bit of Nix knowledge but could yield significant benefits and take a good chunk out of that 20 minutes. Another option might be to use pnpm instead of Yarn and cache your pnpm dependencies. pnpm actually works a bit like Nix in that it creates a pnpm-lock.yaml file with content-based hashes for the full package.json dependency tree. This enables it to quickly determine which parts of the dependency tree it needs to build and which are already available.
- soraminazuki 3y ago
  Nowadays, there's buildNpmPackage [1]. It's included in Nixpkgs, actively maintained, and easier to work with IMO. [1]: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/javascript.section.md#buildnpmpackage-javascript-buildnpmpackage https://github.com/NixOS/nixpkgs/blob/master/doc/languages-f...
duped 3y ago
In order to support resolving dependencies to multiple versions (eg A and B depend on C but at incompatible versions), what npm does is drop the version override for C inside the nested node_modules directory (node_modules/A/node_modules/C, for example). That means that node_modules (as created by a package-lock.json) can't really be cached or be built from caches since it depends on the particular version solution found by npm for a particular project. So there's only so much that Nix can do. It can cache it about as well as using a naive caching scheme with actions/upload-artifact or similar (create a tarball of your node_modules and just cache it across runs, update when you need to). Basically node_modules is inherently large. If you want better performance for caching dependencies use a better programming language environment.

SirensOfTitan 3y ago

Slightly off-topic, but how do folks around here create production packages in nix? I use nix for my dev shells and machine configuration, but haven't yet built production packages using it. More concretely, let's say you have a python backend that uses poetry. Do you just use `poetry install` in your derivation for python-deps? Do you use something like poetry2nix or node2nix and do all of your package management in nix?

grhmc 3y ago
"It depends." poetry2nix is pretty good! Generally, exporting to an OCI image, or targeting a production environment that can run Nix closures natively is the way we go about it. Many of our services use buildLayeredImage[0] and target Fly.io. 0: https://grahamc.com/blog/nix-and-layered-docker-images/ https://grahamc.com/blog/nix-and-layered-docker-images/
- flurie 3y ago
  > targeting a production environment that can run Nix closures natively Do you have some examples for this? The ones that come to mind are things like bare metal/vms with nix, or perhaps disnix, but those are a pretty hard sell over more popular orchestration systems, and I’d like to have more alternatives.
takeda 3y ago
So I so far use it for dev environment and also for CI pipeline (the benefit is that Nix allows caching between runs, it helps with things like: - getting dev environment on CI to be identical to user dev - with minor changes the project is not rebuilt or or rebuilt minimally - the caching works across branches, so for example merging a feature branch to master, if nothing changes the build on master will be very quick I created something similar to nix-cache for gitlab, but I had to create a dedicated runner running NixOS. If I could use NixOS for deployment, at that point I would just point the same binary cache to the machine and use the same derivation to build the app. Because the app was already build by CI, it would just download the compiled version. No need for artifactory or similar. In that scenario (you using poetry) you probably would just use poetry2nix to generate the application. If the OS is not NixOS, but you still want to deploy via nix, then IMO this[2] looks interesting, basically it packages everything in self extracting archive. That you can extract and then run the app. Other alternatives are these bundlers[3], which includes building toArx (works in a way similar to the previous one but pretends everything is in a single file), RPM, DEB, docker (you would have more control over it if you would use the code directly instead of a bundler though) And the last option (probably the most obvious one) is that you can simply just use the tool to build the package. Since you're using poetry, then you can generate a wheel from it. [1] https://github.com/takeda/nix-cde/blob/master/contrib/gitlab-runner/nix-builder.yaml https://github.com/takeda/nix-cde/blob/master/contrib/gitlab... [2] https://github.com/Ninlives/relocatable.nix https://github.com/Ninlives/relocatable.nix [3] https://github.com/NixOS/bundlers/blob/master/flake.nix https://github.com/NixOS/bundlers/blob/master/flake.nix

sgammon 3y ago

this is awesome. i've yet to try Nix, but we are starting a remote cache service and we'd love to be compatible with Nix' caching mechanism we're at https://less.build https://less.build if you want to take a peek -- we will look at adding S3 support! :)

sgammon 3y ago
side note i am happy anytime build caching is on the front page of hckrnews for any reason
a_t48 3y ago
I'm all for faster builds...but how does your product differ from using something like https://ccache.dev/manual/4.7.html#_remote_storage_backends https://ccache.dev/manual/4.7.html#_remote_storage_backends ? (I ask specifically because you advertise being a drop in ccache replacement, or similar)
- sgammon 3y ago
  we are drop-in compatible with ccache, in the sense that you can plug ccache right in (as an HTTP backend), and gain features like authorization policies, analytics, and global caching backed by Cloudflare. that's where we are beginning and then we're working to add spicier features on from there. but first and foremost, we want to serve the ccache/gradle cache crowd with a fantastic protocol-agnostic backend which "just works" and pays for itself in terms of time saved. and thanks for asking :) we are very new
  - a_t48 3y ago
    Ah, cool. Not my usecase, but neat all the same.

wereHamster 3y ago

Just this past week I thought about setting up custom github runners on NixOS machines so that Nix is pre-installed (doesn't need to be installed via a github action) and so that the Nix store can be shared between runs. Though I don't really want to manage the machines, so I might give this new github action a try…

yakshaving_jgt 3y ago
You might be interested in this then. https://github-runners.www-cachix-org.pages.dev/github-runners-for-nix-nixos https://github-runners.www-cachix-org.pages.dev/github-runne...