4 ms·

I use Nix to provide consistent build environments (and then use whatever project tooling). This allows for an even simpler approach. I write a shell.nix that g

I use Nix to provide consistent build environments (and then use whatever project tooling). This allows for an even simpler approach. I write a shell.nix that gives my dev environment. Developers can use that while developing to get a consistent build environment. I can then set up CI in Gitlab by just applying that environment as a before_script. Something like this can be a complete CI file: image: nixos/nix:latest before_script: - nix-env -f shell.nix -i -A buildInputs test_build: stage: test script: - elm make elm-src/Main.elm If you've got loads of dependencies and/or they need to be built and take a while then this can obviously slow down your CI and you'll want to build the docker image. For simpler projects, it is incredibly quick, simple and robust to set up.

ducktective 5y ago
Is it possible to cache the already built dependencies in Github or Gitlab CI if one is using a NixOS image?
- tiew9Vii 5y ago
  I believe so. https://cachix.org/ https://cachix.org/ provides some off the shelf services. There's a few blog posts about using AWS S3 as a binary cache so you can roll your own.
  - danieldk 5y ago
    Cachix is really awesome. There's a few blog posts about using AWS S3 as a binary cache so you can roll your own. Unfortunately, this is really slow due to Nix using Brotli compression at a high compression level. There were some PRs to make the compression method configurable, but the last time I checked, none of them were merged. It is not really an issue if your outputs are small. But I had a private S3 cache and have some derivations that have large output paths, and it made CI excruciatingly slow.
- rkangel 5y ago
  I think of building the docker image as a 'cache' of the dependencies, but actually something like Cachix would provide a more flexible solution (i.e. helping developers as well as CI).
  - ghuntley 5y ago
    Docker is a bad way to handle this. Cachix is the correct solution.
    - takeda 5y ago
      Cachix is definitively the easiest (don't waste my time) solution. If you are on AWS you can also set up S3 bucket.
- takeda 5y ago
  There are multiple caching mechanisms and you can use them together to provide multiple levels. The mentioned cachix (you can also create S3 bucket that can also provide similar functionality). If you set up gitlab runner in NixOS [1], you can follow example configuration which allows to utilize nix local caching. You can of course also use the normal gitlab caching, you will need to ensure the container has /cache mount. [1] https://search.nixos.org/options?channel=21.05&from=0&size=50&sort=relevance&query=gitlab-runner https://search.nixos.org/options?channel=21.05&from=0&size=5...
  - takeda 5y ago
    Here's the gitlab-runner configuration I mentioned in the post (can't edit anymore)[1]. With this you benefit of local caching that nix does. What this means is that if you have a branch and it compiled, and run test. Then as you merge and no source files changed (which happens most of the time), the build on master is instant. [1] https://github.com/NixOS/nixpkgs/blob/ce38fecabe49a76fcd08d03e7562b44b97109aa7/nixos/modules/services/continuous-integration/gitlab-runner.nix#L255-L293 https://github.com/NixOS/nixpkgs/blob/ce38fecabe49a76fcd08d0...

ducktective 5y ago

Is it possible to cache the already built dependencies in Github or Gitlab CI if one is using a NixOS image?

tiew9Vii 5y ago
I believe so. https://cachix.org/ https://cachix.org/ provides some off the shelf services. There's a few blog posts about using AWS S3 as a binary cache so you can roll your own.
- danieldk 5y ago
  Cachix is really awesome. There's a few blog posts about using AWS S3 as a binary cache so you can roll your own. Unfortunately, this is really slow due to Nix using Brotli compression at a high compression level. There were some PRs to make the compression method configurable, but the last time I checked, none of them were merged. It is not really an issue if your outputs are small. But I had a private S3 cache and have some derivations that have large output paths, and it made CI excruciatingly slow.
rkangel 5y ago
I think of building the docker image as a 'cache' of the dependencies, but actually something like Cachix would provide a more flexible solution (i.e. helping developers as well as CI).
- ghuntley 5y ago
  Docker is a bad way to handle this. Cachix is the correct solution.
  - takeda 5y ago
    Cachix is definitively the easiest (don't waste my time) solution. If you are on AWS you can also set up S3 bucket.
takeda 5y ago
There are multiple caching mechanisms and you can use them together to provide multiple levels. The mentioned cachix (you can also create S3 bucket that can also provide similar functionality). If you set up gitlab runner in NixOS [1], you can follow example configuration which allows to utilize nix local caching. You can of course also use the normal gitlab caching, you will need to ensure the container has /cache mount. [1] https://search.nixos.org/options?channel=21.05&from=0&size=50&sort=relevance&query=gitlab-runner https://search.nixos.org/options?channel=21.05&from=0&size=5...
- takeda 5y ago
  Here's the gitlab-runner configuration I mentioned in the post (can't edit anymore)[1]. With this you benefit of local caching that nix does. What this means is that if you have a branch and it compiled, and run test. Then as you merge and no source files changed (which happens most of the time), the build on master is instant. [1] https://github.com/NixOS/nixpkgs/blob/ce38fecabe49a76fcd08d03e7562b44b97109aa7/nixos/modules/services/continuous-integration/gitlab-runner.nix#L255-L293 https://github.com/NixOS/nixpkgs/blob/ce38fecabe49a76fcd08d0...