6 ms·

Nix at work: FlakeHub Cache and private flakes

34 points by grhmc 2y ago

Hey folks, Determinate Systems CEO Graham here. Happy to answer questions about our launch today!

deleted 2y ago
[deleted]
svin_bee 2y ago
Has the situation between you guys and the Nix community cooled down yet? I have honestly avoided NixOS for quite a while because the situation seemed "unstable". I don't particularly care for the complaints brought about by the open letter, I do however worry if a distribution/community is imploding.
- grhmc 2y ago
  There is a lot of turbulance in the project, but I think a lot of that is a natural result of twenty years of history meeting standout commercial success. Our approach here is to continue our pattern of improving Nix and sending patches upstream. We'll also continue improving the ecosystem around Nix, most of that being open source, and only a couple specific projects being proprietary. Regarding the stability of the project: The project and distribution is incredibly vibrant, and not going anywhere. There are tens of thousands of organizations that depend on it every day, and projects that are funded on the time scale of decades. It's here to stay.
  - dandellion 2y ago
    As someone who just went back to Arch after using NixOS as my daily driver for more than six months, what's in dire need of improvement is the documentation, the rest of the ecosystem was all right.
    - marliechiller 2y ago
      I'm still driving nixos, but for me, the stacktraces are the hard part when discerning where and why things have gone wrong
      - grhmc 2y ago
        Ugh. I feel you. I think one thing that makes this hard is how big and intertwined Nixpkgs and NixOS is. All the modules are loaded all the time, and it can make the displayed errors propagate out to somewhere intractably far away from where the error actually is. There's a ton of work still being done in Nix to make that better, and I'm hopeful for an easier future :').
    - grhmc 2y ago
      Agreed. Thank you for saying so. We're working on this in two ways: 1. Reducing the number of moving parts, so there are fewer interfaces that need to be documented and understood ... because they don't exist. 2. Resources like https://zero-to-nix.com/ https://zero-to-nix.com/, and https://docs.determinate.systems/ https://docs.determinate.systems/. This is a big lift, though, so anything we do here in the near future will definitely not be enough.
      - dandellion 2y ago
        Those seem like good steps in the right direction. Other things I'd like to see when I try Nix again would be more consensus of what are good conventions to follow for packages and configurations. Zero-to-Nix looks better than the documentation I used to get started, seems a great resource. But even with the worse documentation I found, I was able to overcome the pretty steep initial learning curve. What did it for me was the lack of documentation for many packages and having to dive into the code every time I needed to configure anything remotely advanced. Whereas in Arch there's always a couple examples that get me 95% of where I want to go, making it very easy to figure out the rest. But thanks for the effort, I'll be looking forward to giving Nix another go in one or two years to see the progress and maybe switch again, as I really liked the whole concept.
  - deng 2y ago
    > There are tens of thousands of organizations that depend on it every day, and projects that are funded on the time scale of decades. It's here to stay. I like Nix, but what turns me off most is this kind of hyperbole from its evangelists. "tens of thousands of organizations" depend on Nix every day? How do you even come up with that kind of number? And as someone who has actually worked on projects that are funded for decades: the first rule is that things are changing, so stay flexible, chose boring technology, and always have a plan for migration. Binding yourself to a niche framework that is also strongly opinionated with an arcane syntax is pretty much the worst you could do. As you say, Nix is already over 20 years old, so yes, it's probably not going anywhere, but that goes both ways: it will probably not vanish in the near future, but it'll probably also not become really popular. And that's fine.
- callahad 2y ago
  I think it's worth jumping in. Fifteen years ago, it wasn't clear whether Git, Mercurial, or Bazaar would win, but it was evident that something fundamental had forever changed in revision control systems. Nix feels similar, but for build systems, package management, and configuration management all at once.
  - dietr1ch 2y ago
    Right, I don't know if nix will manage to evolve, or a new implementation will come up, but the box with the wonders of nix has been already open and there's no way people will want to go back to mutable and non-deterministic systems.
cloudripper 2y ago
Given the extensive infrastructure you and others are building around flakes in the Nix ecosystem, despite flakes still being experimental, what does the roadmap look like for your efforts, and more broadly for officially establishing flakes??
- grhmc 2y ago
  Sure. To start with, we see flakes as stable as they are today: https://determinate.systems/posts/experimental-does-not-mean-unstable/ https://determinate.systems/posts/experimental-does-not-mean.... The ecosystem has broadly adopted flakes as their preferred way to use Nix, and we're standing by them and are committing to not breaking their flakes in the future. On Monday, we launched Determinate Nix to help make this promise more real: https://determinate.systems/posts/announcing-determinate-nix/ https://determinate.systems/posts/announcing-determinate-nix.... Note: Determinate Nix is not a fork, it is a downstream. Our plan, and intent, is to keep all our patches sent to the upstream project first.
  - slabity 2y ago
    > Note: Determinate Nix is not a fork, it is a downstream. Our plan, and intent, is to keep all our patches sent to the upstream project first. And what happens if the Nix community doesn't pull those patches, and instead goes with a different solution? Will your downstream adapt to the upstream project, possibly breaking things for your customers?
    - grhmc 2y ago
      We won't break our customers. Indeed, part of the motivation for our downstream distribution is to be able to ship some of our patches faster than upstream wants to. However, these patches are generally about usability improvements that are not incompatible. If the upstream project evolves in a different direction, it will be on us to move with them too.
- depr 2y ago
  The roadmap is closed-source solutions as described here: https://discourse.nixos.org/t/introducing-flakehub/32044/3 https://discourse.nixos.org/t/introducing-flakehub/32044/3
deleted 2y ago
[deleted]
edem 2y ago
Can someone do a ELI5 of Nix and flakes for me? I've looked at the docs numerous times and bounced back each time after I failed to understand the USP of Nix...I just don't know what's the main use case(s).
- biggestlou 2y ago
  I've been meaning to write something pithy on this and this is a great spur to action. Basically Nix is something you use to build software reproducibly, that is, where you can take the same inputs and get the exact same bytes on disk every time you build it. The "software" here could be something simple-ish like a CLI tool or a UNIX utility, and it could be as complex as an entire functioning Linux system, and it could be something that you usually build using other tools, like Docker or Podman containers. Nix can do all of this. I'd say that the things people most often use Nix for are: 1. Project-specific development environments. You want everyone on your team to have the exact same version of Python, jq, Terraform, Cargo, and Bun in a specific project? Nix can do that. 2. Home environments. You want to replace your dotfiles and Homebrew and apt-get with something declarative that you have an actual language for? Nix can do that. So yeah, development environments, home environments, whole Linux systems, OCI containers, standard packages (like CLI tools), things like VS Code extension, Neovim plugins... all of this stuff can be built using Nix. Pretty powerful stuff.
  - edem 2y ago
    now that makes a lot of sense. in fact id like to have both! does this work in wsl?
    - grhmc 2y ago
      Yes! 1. Enable systemd: https://devblogs.microsoft.com/commandline/systemd-support-is-now-available-in-wsl/#how-can-you-get-systemd-on-your-machine https://devblogs.microsoft.com/commandline/systemd-support-i... 2. Install as normal: curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix https://install.determinate.systems/nix | sh -s -- install --determinate 3. https://zero-to-nix.com/ https://zero-to-nix.com/ :)
jamies 2y ago
I appreciate someone from the project commenting here. I left Nix because it just has too many warts, and unfortunately, they've only increased over time, which is the opposite of what I expected. - I'd update Nixpkgs, and my build would stop working. The logging was awful, so fixing it took hours. - The mono repo sounds nice, but selective updates are almost impossible. - Building our Go project in a Nix package was nearly impossible with private GitHub repos, so we moved on. - Bash scripts never work because binaries are never where they should be in Linux. I could go on, but my experience with Nix and NixOS left me with significant scars. I’m hopeful that someone will eventually create a more user-friendly deterministic OS.
- grhmc 2y ago
  > - Building our Go project in a Nix package was nearly impossible with private GitHub repos, so we moved on. Yes. This is a nightmare, and is the literal step-one problem FlakeHub is designed to solve. Regarding the monorepo causing big-bang updates that broke all your stuff: continued, strong, agreement. I think as nixpkgs continues to grow, and usage grows, we'll see a wider adoption of flakes from a pure "how do we all work together?" perspective. That's the flip side of FlakeHub's initial concept.
  - jamies 2y ago
    Please keep at it! Doing what GitHub did for Git is a real opportunity I think for Nix. We'll be keeping an eye on Determinate's work...
    - grhmc 2y ago
      Thank you, we will! =). I think there is a lot of opportunity here around a cohesive workflow around Nix. That is a big part about what I was consulting on for so many years. Maybe check out our demo on our home page: https://determinate.systems/ https://determinate.systems/ -- you might like it.
- gaganyaan 2y ago
  Not sure if you know this, but this is the way to add dependencies to bash scripts: https://nixos.wiki/wiki/Nix-shell_shebang https://nixos.wiki/wiki/Nix-shell_shebang It should add everything to $PATH, so unless you've got hardcoded bin paths, there shouldn't be any problems.
0cf8612b2e1e 2y ago
I love the idea of Nix, but the current implementation seems too impossibly complex to ever go mainstream. Dynamic hard-to-debug DSL, loads of novel concepts, difficult to do a piecemeal transition, etc. One poster in this thread reported how their supposedly immutable system configuration broke with an update. Is there anything on the horizon which could embrace the ideas in a better package? Does Guix smooth over enough rough edges?
- grhmc 2y ago
  I believe a lot of this comes from Nix being largely "low policy", meaning you can do anything, any way you want, you just have to figure out how. Our goal is to make a more high-policy version that has more workflow and "rails" out of the box to make a good experience.

34 points by grhmc 2y ago

Hey folks, Determinate Systems CEO Graham here. Happy to answer questions about our launch today!

deleted 2y ago

[deleted]

svin_bee 2y ago

Has the situation between you guys and the Nix community cooled down yet? I have honestly avoided NixOS for quite a while because the situation seemed "unstable". I don't particularly care for the complaints brought about by the open letter, I do however worry if a distribution/community is imploding.

grhmc 2y ago
There is a lot of turbulance in the project, but I think a lot of that is a natural result of twenty years of history meeting standout commercial success. Our approach here is to continue our pattern of improving Nix and sending patches upstream. We'll also continue improving the ecosystem around Nix, most of that being open source, and only a couple specific projects being proprietary. Regarding the stability of the project: The project and distribution is incredibly vibrant, and not going anywhere. There are tens of thousands of organizations that depend on it every day, and projects that are funded on the time scale of decades. It's here to stay.
- dandellion 2y ago
  As someone who just went back to Arch after using NixOS as my daily driver for more than six months, what's in dire need of improvement is the documentation, the rest of the ecosystem was all right.
  - marliechiller 2y ago
    I'm still driving nixos, but for me, the stacktraces are the hard part when discerning where and why things have gone wrong
    - grhmc 2y ago
      Ugh. I feel you. I think one thing that makes this hard is how big and intertwined Nixpkgs and NixOS is. All the modules are loaded all the time, and it can make the displayed errors propagate out to somewhere intractably far away from where the error actually is. There's a ton of work still being done in Nix to make that better, and I'm hopeful for an easier future :').
  - grhmc 2y ago
    Agreed. Thank you for saying so. We're working on this in two ways: 1. Reducing the number of moving parts, so there are fewer interfaces that need to be documented and understood ... because they don't exist. 2. Resources like https://zero-to-nix.com/ https://zero-to-nix.com/, and https://docs.determinate.systems/ https://docs.determinate.systems/. This is a big lift, though, so anything we do here in the near future will definitely not be enough.
    - dandellion 2y ago
      Those seem like good steps in the right direction. Other things I'd like to see when I try Nix again would be more consensus of what are good conventions to follow for packages and configurations. Zero-to-Nix looks better than the documentation I used to get started, seems a great resource. But even with the worse documentation I found, I was able to overcome the pretty steep initial learning curve. What did it for me was the lack of documentation for many packages and having to dive into the code every time I needed to configure anything remotely advanced. Whereas in Arch there's always a couple examples that get me 95% of where I want to go, making it very easy to figure out the rest. But thanks for the effort, I'll be looking forward to giving Nix another go in one or two years to see the progress and maybe switch again, as I really liked the whole concept.
- deng 2y ago
  > There are tens of thousands of organizations that depend on it every day, and projects that are funded on the time scale of decades. It's here to stay. I like Nix, but what turns me off most is this kind of hyperbole from its evangelists. "tens of thousands of organizations" depend on Nix every day? How do you even come up with that kind of number? And as someone who has actually worked on projects that are funded for decades: the first rule is that things are changing, so stay flexible, chose boring technology, and always have a plan for migration. Binding yourself to a niche framework that is also strongly opinionated with an arcane syntax is pretty much the worst you could do. As you say, Nix is already over 20 years old, so yes, it's probably not going anywhere, but that goes both ways: it will probably not vanish in the near future, but it'll probably also not become really popular. And that's fine.
callahad 2y ago
I think it's worth jumping in. Fifteen years ago, it wasn't clear whether Git, Mercurial, or Bazaar would win, but it was evident that something fundamental had forever changed in revision control systems. Nix feels similar, but for build systems, package management, and configuration management all at once.
- dietr1ch 2y ago
  Right, I don't know if nix will manage to evolve, or a new implementation will come up, but the box with the wonders of nix has been already open and there's no way people will want to go back to mutable and non-deterministic systems.

cloudripper 2y ago

Given the extensive infrastructure you and others are building around flakes in the Nix ecosystem, despite flakes still being experimental, what does the roadmap look like for your efforts, and more broadly for officially establishing flakes??

grhmc 2y ago
Sure. To start with, we see flakes as stable as they are today: https://determinate.systems/posts/experimental-does-not-mean-unstable/ https://determinate.systems/posts/experimental-does-not-mean.... The ecosystem has broadly adopted flakes as their preferred way to use Nix, and we're standing by them and are committing to not breaking their flakes in the future. On Monday, we launched Determinate Nix to help make this promise more real: https://determinate.systems/posts/announcing-determinate-nix/ https://determinate.systems/posts/announcing-determinate-nix.... Note: Determinate Nix is not a fork, it is a downstream. Our plan, and intent, is to keep all our patches sent to the upstream project first.
- slabity 2y ago
  > Note: Determinate Nix is not a fork, it is a downstream. Our plan, and intent, is to keep all our patches sent to the upstream project first. And what happens if the Nix community doesn't pull those patches, and instead goes with a different solution? Will your downstream adapt to the upstream project, possibly breaking things for your customers?
  - grhmc 2y ago
    We won't break our customers. Indeed, part of the motivation for our downstream distribution is to be able to ship some of our patches faster than upstream wants to. However, these patches are generally about usability improvements that are not incompatible. If the upstream project evolves in a different direction, it will be on us to move with them too.
depr 2y ago
The roadmap is closed-source solutions as described here: https://discourse.nixos.org/t/introducing-flakehub/32044/3 https://discourse.nixos.org/t/introducing-flakehub/32044/3

deleted 2y ago

[deleted]

edem 2y ago

Can someone do a ELI5 of Nix and flakes for me? I've looked at the docs numerous times and bounced back each time after I failed to understand the USP of Nix...I just don't know what's the main use case(s).

biggestlou 2y ago
I've been meaning to write something pithy on this and this is a great spur to action. Basically Nix is something you use to build software reproducibly, that is, where you can take the same inputs and get the exact same bytes on disk every time you build it. The "software" here could be something simple-ish like a CLI tool or a UNIX utility, and it could be as complex as an entire functioning Linux system, and it could be something that you usually build using other tools, like Docker or Podman containers. Nix can do all of this. I'd say that the things people most often use Nix for are: 1. Project-specific development environments. You want everyone on your team to have the exact same version of Python, jq, Terraform, Cargo, and Bun in a specific project? Nix can do that. 2. Home environments. You want to replace your dotfiles and Homebrew and apt-get with something declarative that you have an actual language for? Nix can do that. So yeah, development environments, home environments, whole Linux systems, OCI containers, standard packages (like CLI tools), things like VS Code extension, Neovim plugins... all of this stuff can be built using Nix. Pretty powerful stuff.
- edem 2y ago
  now that makes a lot of sense. in fact id like to have both! does this work in wsl?
  - grhmc 2y ago
    Yes! 1. Enable systemd: https://devblogs.microsoft.com/commandline/systemd-support-is-now-available-in-wsl/#how-can-you-get-systemd-on-your-machine https://devblogs.microsoft.com/commandline/systemd-support-i... 2. Install as normal: curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix https://install.determinate.systems/nix | sh -s -- install --determinate 3. https://zero-to-nix.com/ https://zero-to-nix.com/ :)

jamies 2y ago

I appreciate someone from the project commenting here. I left Nix because it just has too many warts, and unfortunately, they've only increased over time, which is the opposite of what I expected. - I'd update Nixpkgs, and my build would stop working. The logging was awful, so fixing it took hours. - The mono repo sounds nice, but selective updates are almost impossible. - Building our Go project in a Nix package was nearly impossible with private GitHub repos, so we moved on. - Bash scripts never work because binaries are never where they should be in Linux. I could go on, but my experience with Nix and NixOS left me with significant scars. I’m hopeful that someone will eventually create a more user-friendly deterministic OS.

grhmc 2y ago
> - Building our Go project in a Nix package was nearly impossible with private GitHub repos, so we moved on. Yes. This is a nightmare, and is the literal step-one problem FlakeHub is designed to solve. Regarding the monorepo causing big-bang updates that broke all your stuff: continued, strong, agreement. I think as nixpkgs continues to grow, and usage grows, we'll see a wider adoption of flakes from a pure "how do we all work together?" perspective. That's the flip side of FlakeHub's initial concept.
- jamies 2y ago
  Please keep at it! Doing what GitHub did for Git is a real opportunity I think for Nix. We'll be keeping an eye on Determinate's work...
  - grhmc 2y ago
    Thank you, we will! =). I think there is a lot of opportunity here around a cohesive workflow around Nix. That is a big part about what I was consulting on for so many years. Maybe check out our demo on our home page: https://determinate.systems/ https://determinate.systems/ -- you might like it.
gaganyaan 2y ago
Not sure if you know this, but this is the way to add dependencies to bash scripts: https://nixos.wiki/wiki/Nix-shell_shebang https://nixos.wiki/wiki/Nix-shell_shebang It should add everything to $PATH, so unless you've got hardcoded bin paths, there shouldn't be any problems.

0cf8612b2e1e 2y ago

I love the idea of Nix, but the current implementation seems too impossibly complex to ever go mainstream. Dynamic hard-to-debug DSL, loads of novel concepts, difficult to do a piecemeal transition, etc. One poster in this thread reported how their supposedly immutable system configuration broke with an update. Is there anything on the horizon which could embrace the ideas in a better package? Does Guix smooth over enough rough edges?

grhmc 2y ago
I believe a lot of this comes from Nix being largely "low policy", meaning you can do anything, any way you want, you just have to figure out how. Our goal is to make a more high-policy version that has more workflow and "rails" out of the box to make a good experience.