5 ms·

Improved evaluation times with pre-resolved Nix store paths

41 points by biggestlou 2y ago

_cenw 2y ago
DetSys, once again, building a moat. I really do not like how this approach depends on their seemingly not open source backend. Nix needs tools like this, no question, but I'd hate to see it get adoption through a way that isn't reproducible by anyone. It looks like it could be a repeat of the Snap store problem - interesting tech, made unusable through de-facto dependencies on proprietary services.
- mongol 2y ago
  Is it really a moat though? The same flake can still be evaluated without this service, it just takes longer.
  - _cenw 2y ago
    And I can still install a snap on an Ubuntu Core machine by building my own distribution method, but it will be a downgrade from the experience of running "snap install something" that introduced me to it. So nobody does. Lix (yes, I will mention it again) essentially exists by a large degree because the overlap between the people that control Nix and DetSys is big, even very big if you look at CppNix.
    - mongol 2y ago
      I don't follow this argument. Unless you chose to step in to DetSys ecosystem with this fh tool or whatever, this has no impact on you at all.
    - otabdeveloper4 2y ago
      > exists by a large degree because the overlap between the people that control Nix and DetSys is big No, it exists because of sexual identity politics. 11 out of 12 of the Lix developer team are transsexuals. Clearly the selection isn't about technical issues.
      - _cenw 2y ago
        Who's doing the identity politics now.
        anonfordays 2y ago
        Clearly the Lix team.
        nothrabannosir 2y ago
        Ok but they’re creating a good product, in their “own” space, not bothering anyone, and they increase competition which benefits the entire ecosystem, so to be frank… who cares? Lix is a net win for everyone. It would behoove us to stop taking pot shots at them because if you want nix to succeed, the presence of Lix is a net positive.
        otabdeveloper4 2y ago
        Not sure deliberate and obtuse fragmentation benefits anyone. If they started to fix some of the long-standing Nix implementation issues (namely, evaluation performance) that would be one thing. But what they actually did is rage-fork nixpkgs and register new domains.
        anonfordays 2y ago
        >Ok but they’re creating a good product, Debatable. >in their “own” space, not bothering anyone, and they increase competition which benefits the entire ecosystem, so to be frank… who cares? Sounds like White supremacy and Neo-Nazism. Why not say that about every other project that leftist enter and try to push their religious beliefs? >Lix is a net win for everyone. Most definitely not, fragmentation and religious proselytizing destroys projects. >It would behoove us to stop taking pot shots at them The truth shouldn't be hidden, regardless if they're "pot shots" or not. >want nix to succeed, the presence of Lix is a net positive. The Lix team are the ones that destroyed the nix community with their totalitarianism, leftist/woke ideology, and subversive actions. They are a net negative.
- larusso 2y ago
  I didn‘t look too deep into Nix the last couple of months (> 12) and was wondering while reading: what the hell is fh now. Another abstraction? I share your views here!
  - biggestlou 2y ago
    fh is the CLI tool for FlakeHub. It's been around almost as long as FlakeHub itself. It's pretty standard fare, not really an abstraction per se.
    - rounce 2y ago
      ‘Standard’ if you live in the flakehub ecosystem, which the vast majority of Nix users don’t.
      - biggestlou 2y ago
        Standard in the sense of "a non-magical CLI tool that wraps a platform's HTTP API," not standard in the sense of "used by a majority of users."
- biggestlou 2y ago
  Out of curiosity, do you object to the existence of services like Cachix, Nixbuild, and Garnix, which also do not have open source backends?
  - _cenw 2y ago
    If their founder was the person with principal control over what goes into CppNix, yes.
    - biggestlou 2y ago
      Is there a Nix-based company co-founded by Eelco Dolstra to which you wouldn't object?
      - _cenw 2y ago
        Sure, but you seem to sidestep the primary accusation that Nix itself got neglected in favor of DetSys. Which is most certainly true for at least the installer.
        biggestlou 2y ago
        The expectation seems to be that Eelco work on Nix full time for free lest he be accused of neglect. Is there a way for Eelco to make money that you would approve of?
        _cenw 2y ago
        I would've replied to you but HN's intransparent timeout mechanism hit me. I'd invite everyone to wonder why Lix exists (and is better enough to be used in some parts of Nix's CI now), why CppNix stable was stuck 6 versions behind on nixpkgs until half a year ago, and what kept Lix contributors from working on CppNix instead. Tip: "I want free labor from Eelco" is not the answer.
        justinrubek 2y ago
        Nobody said this. I suggest that you not escalate the discussion in this direction. You're obviously biased given involvement with detsys, nobody expects you to think differently. This crass dismissal off the issue with a strawman isn't right.
- anonfordays 2y ago
  They're not building a moat, they were excommunicated and banished. Zealots hate to see heretics living successful lives. More power to them.
  - _cenw 2y ago
    I think you may be mixing up some things. Anyway, where's this sudden influx of traffic coming from?
    - anonfordays 2y ago
      >I think you may be mixing up some things. I don't think so. >Anyway, where's this sudden influx of traffic coming from? What traffic?
- weitzj 2y ago
  Love what they are doing. At least you get the chance to introduce Nix in the enterprise with the MacOS installer, having figured out private CAs and the MacOS keychain for example. Then MDM.
biggestlou 2y ago
Blog post author here! Please feel free to ask me any questions directly in this forum :)
- mongol 2y ago
  Can the evaluation tax be quantified in some way? In other words, how big is the problem that this solves?
  - biggestlou 2y ago
    Unfortunately, it's really hard to generalize. If the evaluation involves Nixpkgs, for example, Nix needs to copy Nixpkgs into the Nix store, which is many MB. And of course this needs to happen for all flake inputs. I've seen this evaluation take a few seconds and I've seen it take several minutes, but in general the more flake inputs the longer you can expect it to take.
    - q3k 2y ago
      > If the evaluation involves Nixpkgs, for example, Nix needs to copy Nixpkgs into the Nix store, which is many MB. Only if you use flakes (and this behavior is a good reason to not use them).
      - biggestlou 2y ago
        Using flakes or not, any store path evaluation involving Nixpkgs is going to be computationally intensive, potentially too intensive for a memory-constrained device.
  - grhmc 2y ago
    (DetSys Cofounder here) I've had Raspberry Pi's give up their last ghostly breath evaluating NixOS.
mongol 2y ago
This will make discourse.nixos.org explode again. It happens every time Determinate System publishes something.
- _huayra_ 2y ago
  The unusually high amount of drama in NixOS is one of the reasons I have been hesitant to start using it. Jon Ringer (a former contributor) posted a giant video [0] outlining his history with the project and how he left / was forced out (amongst several other key folks). For the record, I don't know how to characterize his exit, only that there was an absolute ton of drama around it that wasn't really related to even any technical aspect. [0] https://www.youtube.com/watch?v=gp0FI8Gw1iA https://www.youtube.com/watch?v=gp0FI8Gw1iA
  - mongol 2y ago
    Listened to this video yesterday and it just makes me sad. In my opinion, Jon has been let down by the community to which he belonged. It is clear there were differences of opinion and so on, but I did not see anything that motivated his expulsion permanently. It left an especially bad taste that it was on recommendation by the Constitutional Assembly. Their mission was to propose a constitution, not to act as arbiter in moderation matters.
  - _cenw 2y ago
    I don't agree with Jon on a personal level, but who was technically in the right aside, he also made some choices that'd accelerate your exit from any organization or community during the events. Let's just say his approach wasn't one of reconciliation and de-escalation.
  - RGamma 2y ago
    What a collective waste of energy. Had no idea any of this was/is going on. For the sake of my personal systems I hope NixOS makes it; wouldn't be the first community that fails because of such kindergarten nonsense...
    - 2y ago
knoopx 2y ago
what a coincidence, my eval times have worsened over the past months too!
deleted 2y ago
[deleted]
jljljl 2y ago
This is cool! We do something similar in Devbox where we pre-evaluate store paths from the Nix repository, and then fetch the packages directly from the official Nix cache. This greatly reduces the evaluation time. Here's a blog post we wrote on the topic: https://www.jetify.com/blog/how-we-sped-up-nix-package-installs-in-devbox https://www.jetify.com/blog/how-we-sped-up-nix-package-insta...
- jkarni 2y ago
  We also do something similar at garnix, but when enabling incremental builds. Instead of just skipping the build stage, we also “normalize” the eval into just the store path, and skipping it the second time around. Mentioned in passing in https://garnix.io/blog/incremental-builds https://garnix.io/blog/incremental-builds. This is even more significant because in this case you might otherwise be eval-ing several layers of flakes.
pinetroey 2y ago
Is this similar to, using nixos-rebuild and target a second machine? I use my laptop's nixos-rebuild and push a new generation to another device. The 2nd device doesn't have the config, so it won't be able to build it. For these devices, I don't want the config to be on the 2nd device.

41 points by biggestlou 2y ago

DetSys, once again, building a moat. I really do not like how this approach depends on their seemingly not open source backend. Nix needs tools like this, no question, but I'd hate to see it get adoption through a way that isn't reproducible by anyone. It looks like it could be a repeat of the Snap store problem - interesting tech, made unusable through de-facto dependencies on proprietary services.

mongol 2y ago
Is it really a moat though? The same flake can still be evaluated without this service, it just takes longer.
- _cenw 2y ago
  And I can still install a snap on an Ubuntu Core machine by building my own distribution method, but it will be a downgrade from the experience of running "snap install something" that introduced me to it. So nobody does. Lix (yes, I will mention it again) essentially exists by a large degree because the overlap between the people that control Nix and DetSys is big, even very big if you look at CppNix.
  - mongol 2y ago
    I don't follow this argument. Unless you chose to step in to DetSys ecosystem with this fh tool or whatever, this has no impact on you at all.
  - otabdeveloper4 2y ago
    > exists by a large degree because the overlap between the people that control Nix and DetSys is big No, it exists because of sexual identity politics. 11 out of 12 of the Lix developer team are transsexuals. Clearly the selection isn't about technical issues.
    - _cenw 2y ago
      Who's doing the identity politics now.
      - anonfordays 2y ago
        Clearly the Lix team.
        nothrabannosir 2y ago
        Ok but they’re creating a good product, in their “own” space, not bothering anyone, and they increase competition which benefits the entire ecosystem, so to be frank… who cares? Lix is a net win for everyone. It would behoove us to stop taking pot shots at them because if you want nix to succeed, the presence of Lix is a net positive.
        otabdeveloper4 2y ago
        Not sure deliberate and obtuse fragmentation benefits anyone. If they started to fix some of the long-standing Nix implementation issues (namely, evaluation performance) that would be one thing. But what they actually did is rage-fork nixpkgs and register new domains.
        anonfordays 2y ago
        >Ok but they’re creating a good product, Debatable. >in their “own” space, not bothering anyone, and they increase competition which benefits the entire ecosystem, so to be frank… who cares? Sounds like White supremacy and Neo-Nazism. Why not say that about every other project that leftist enter and try to push their religious beliefs? >Lix is a net win for everyone. Most definitely not, fragmentation and religious proselytizing destroys projects. >It would behoove us to stop taking pot shots at them The truth shouldn't be hidden, regardless if they're "pot shots" or not. >want nix to succeed, the presence of Lix is a net positive. The Lix team are the ones that destroyed the nix community with their totalitarianism, leftist/woke ideology, and subversive actions. They are a net negative.
larusso 2y ago
I didn‘t look too deep into Nix the last couple of months (> 12) and was wondering while reading: what the hell is fh now. Another abstraction? I share your views here!
- biggestlou 2y ago
  fh is the CLI tool for FlakeHub. It's been around almost as long as FlakeHub itself. It's pretty standard fare, not really an abstraction per se.
  - rounce 2y ago
    ‘Standard’ if you live in the flakehub ecosystem, which the vast majority of Nix users don’t.
    - biggestlou 2y ago
      Standard in the sense of "a non-magical CLI tool that wraps a platform's HTTP API," not standard in the sense of "used by a majority of users."
biggestlou 2y ago
Out of curiosity, do you object to the existence of services like Cachix, Nixbuild, and Garnix, which also do not have open source backends?
- _cenw 2y ago
  If their founder was the person with principal control over what goes into CppNix, yes.
  - biggestlou 2y ago
    Is there a Nix-based company co-founded by Eelco Dolstra to which you wouldn't object?
    - _cenw 2y ago
      Sure, but you seem to sidestep the primary accusation that Nix itself got neglected in favor of DetSys. Which is most certainly true for at least the installer.
      - biggestlou 2y ago
        The expectation seems to be that Eelco work on Nix full time for free lest he be accused of neglect. Is there a way for Eelco to make money that you would approve of?
        _cenw 2y ago
        I would've replied to you but HN's intransparent timeout mechanism hit me. I'd invite everyone to wonder why Lix exists (and is better enough to be used in some parts of Nix's CI now), why CppNix stable was stuck 6 versions behind on nixpkgs until half a year ago, and what kept Lix contributors from working on CppNix instead. Tip: "I want free labor from Eelco" is not the answer.
        justinrubek 2y ago
        Nobody said this. I suggest that you not escalate the discussion in this direction. You're obviously biased given involvement with detsys, nobody expects you to think differently. This crass dismissal off the issue with a strawman isn't right.
anonfordays 2y ago
They're not building a moat, they were excommunicated and banished. Zealots hate to see heretics living successful lives. More power to them.
- _cenw 2y ago
  I think you may be mixing up some things. Anyway, where's this sudden influx of traffic coming from?
  - anonfordays 2y ago
    >I think you may be mixing up some things. I don't think so. >Anyway, where's this sudden influx of traffic coming from? What traffic?
weitzj 2y ago
Love what they are doing. At least you get the chance to introduce Nix in the enterprise with the MacOS installer, having figured out private CAs and the MacOS keychain for example. Then MDM.

biggestlou 2y ago

Blog post author here! Please feel free to ask me any questions directly in this forum :)

mongol 2y ago
Can the evaluation tax be quantified in some way? In other words, how big is the problem that this solves?
- biggestlou 2y ago
  Unfortunately, it's really hard to generalize. If the evaluation involves Nixpkgs, for example, Nix needs to copy Nixpkgs into the Nix store, which is many MB. And of course this needs to happen for all flake inputs. I've seen this evaluation take a few seconds and I've seen it take several minutes, but in general the more flake inputs the longer you can expect it to take.
  - q3k 2y ago
    > If the evaluation involves Nixpkgs, for example, Nix needs to copy Nixpkgs into the Nix store, which is many MB. Only if you use flakes (and this behavior is a good reason to not use them).
    - biggestlou 2y ago
      Using flakes or not, any store path evaluation involving Nixpkgs is going to be computationally intensive, potentially too intensive for a memory-constrained device.
- grhmc 2y ago
  (DetSys Cofounder here) I've had Raspberry Pi's give up their last ghostly breath evaluating NixOS.

mongol 2y ago

This will make discourse.nixos.org explode again. It happens every time Determinate System publishes something.

_huayra_ 2y ago
The unusually high amount of drama in NixOS is one of the reasons I have been hesitant to start using it. Jon Ringer (a former contributor) posted a giant video [0] outlining his history with the project and how he left / was forced out (amongst several other key folks). For the record, I don't know how to characterize his exit, only that there was an absolute ton of drama around it that wasn't really related to even any technical aspect. [0] https://www.youtube.com/watch?v=gp0FI8Gw1iA https://www.youtube.com/watch?v=gp0FI8Gw1iA
- mongol 2y ago
  Listened to this video yesterday and it just makes me sad. In my opinion, Jon has been let down by the community to which he belonged. It is clear there were differences of opinion and so on, but I did not see anything that motivated his expulsion permanently. It left an especially bad taste that it was on recommendation by the Constitutional Assembly. Their mission was to propose a constitution, not to act as arbiter in moderation matters.
- _cenw 2y ago
  I don't agree with Jon on a personal level, but who was technically in the right aside, he also made some choices that'd accelerate your exit from any organization or community during the events. Let's just say his approach wasn't one of reconciliation and de-escalation.
- RGamma 2y ago
  What a collective waste of energy. Had no idea any of this was/is going on. For the sake of my personal systems I hope NixOS makes it; wouldn't be the first community that fails because of such kindergarten nonsense...
  - 2y ago

knoopx 2y ago

what a coincidence, my eval times have worsened over the past months too!

deleted 2y ago

[deleted]

jljljl 2y ago

This is cool! We do something similar in Devbox where we pre-evaluate store paths from the Nix repository, and then fetch the packages directly from the official Nix cache. This greatly reduces the evaluation time. Here's a blog post we wrote on the topic: https://www.jetify.com/blog/how-we-sped-up-nix-package-installs-in-devbox https://www.jetify.com/blog/how-we-sped-up-nix-package-insta...

jkarni 2y ago
We also do something similar at garnix, but when enabling incremental builds. Instead of just skipping the build stage, we also “normalize” the eval into just the store path, and skipping it the second time around. Mentioned in passing in https://garnix.io/blog/incremental-builds https://garnix.io/blog/incremental-builds. This is even more significant because in this case you might otherwise be eval-ing several layers of flakes.

pinetroey 2y ago

Is this similar to, using nixos-rebuild and target a second machine? I use my laptop's nixos-rebuild and push a new generation to another device. The 2nd device doesn't have the config, so it won't be able to build it. For these devices, I don't want the config to be on the 2nd device.