5 ms·

How does the Magic Nix Cache know which parts of my build are deterministic and which aren't? I suppose maybe it will only work if I split my build up into mul

How does the Magic Nix Cache know which parts of my build are deterministic and which aren't? I suppose maybe it will only work if I split my build up into multiple steps such that Nix will know to skip those first steps. If Nix knows that, I suppose the Magic Nix Cache also knows?

biggestlou 3y ago
Could you possibly clarify what you mean by "skip those first steps?" Which steps are you referring to?
- iso8859-1 3y ago
  Presumably, the Magic Nix Cache speeds up your build by caching the output of the "first steps" (really leaves in a tree of inputs). Otherwise, how would it work? So by steps, I am referring to the steps that your Nix derivation consists of.
  - biggestlou 3y ago
    When Nix "realises" a derivation, it realises the entire dependency tree (here, "realises" means either building or fetching, depending on whether a dependency is already in the Nix store). For every single derivation (dependency) in the tree, Nix first calculates what the store path for that dependency would be and then uses that to determine if it's already stored in the Nix store. So it would look at, say, a glibc dependency in a derivation and determine that the Nix store path would be /nix/store/7kn2mkg0g49lfflkdip7i39q3zsck4pc-glibc. If that's already in the Nix store then it doesn't need to build that. And it applies this logic throughout the entire dependency tree. In some cases, the entire dependency tree has already been built and written to the Nix store, in which Nix knows that it doesn't have to build anything. So Nix's caching logic doesn't apply only to the "first steps" of a Nix build (or realisation, to be more specific); it applies to all steps.
    - grhmc 3y ago
      Check out https://zero-to-nix.com/concepts/closures https://zero-to-nix.com/concepts/closures and https://zero-to-nix.com/concepts/realisation https://zero-to-nix.com/concepts/realisation, which I think add more color to this (and also is largely authored by biggestlou here.)
grhmc 3y ago
Nix's design gives good guarantees about reproducibility. It may not be bit-perfect, but in general it is _very_ good. Maybe the Zero to Nix article on Caching, and its linked pages will help? https://zero-to-nix.com/concepts/caching https://zero-to-nix.com/concepts/caching The long and short of it is a merkle tree of hashed inputs :).
takeda 3y ago
Nix builds things in sandbox (no network access, no file system access etc) this can make creating derivation for application that has build with side effects a frustrating experience, but that brings reproducibility (or at least gets us really close there) so generally this is not a concern. As for splitting build into steps. I am not entirely sure what are you trying to do, but you should not need to do that. Each package in Nix has a hash, which is generated from things like hash of the source code, of the dependencies, compile flags, system architecture etc. This means that you could have multiple versions of the same package in /nix/store with different compile options or different dependencies. When you need a given dependency you know all the information to generate the hash and can easily know if you need to build it or you can use cached version. This is what makes binary cache a pretty much plug and play and don't need to worry what files to cache or whether you should split build into stages.
aidenn0 3y ago
You tell it; when you define a derivation, you are promising it is at least logically deterministic.

biggestlou 3y ago

Could you possibly clarify what you mean by "skip those first steps?" Which steps are you referring to?

iso8859-1 3y ago
Presumably, the Magic Nix Cache speeds up your build by caching the output of the "first steps" (really leaves in a tree of inputs). Otherwise, how would it work? So by steps, I am referring to the steps that your Nix derivation consists of.
- biggestlou 3y ago
  When Nix "realises" a derivation, it realises the entire dependency tree (here, "realises" means either building or fetching, depending on whether a dependency is already in the Nix store). For every single derivation (dependency) in the tree, Nix first calculates what the store path for that dependency would be and then uses that to determine if it's already stored in the Nix store. So it would look at, say, a glibc dependency in a derivation and determine that the Nix store path would be /nix/store/7kn2mkg0g49lfflkdip7i39q3zsck4pc-glibc. If that's already in the Nix store then it doesn't need to build that. And it applies this logic throughout the entire dependency tree. In some cases, the entire dependency tree has already been built and written to the Nix store, in which Nix knows that it doesn't have to build anything. So Nix's caching logic doesn't apply only to the "first steps" of a Nix build (or realisation, to be more specific); it applies to all steps.
  - grhmc 3y ago
    Check out https://zero-to-nix.com/concepts/closures https://zero-to-nix.com/concepts/closures and https://zero-to-nix.com/concepts/realisation https://zero-to-nix.com/concepts/realisation, which I think add more color to this (and also is largely authored by biggestlou here.)

grhmc 3y ago

Nix's design gives good guarantees about reproducibility. It may not be bit-perfect, but in general it is _very_ good. Maybe the Zero to Nix article on Caching, and its linked pages will help? https://zero-to-nix.com/concepts/caching https://zero-to-nix.com/concepts/caching The long and short of it is a merkle tree of hashed inputs :).

takeda 3y ago

Nix builds things in sandbox (no network access, no file system access etc) this can make creating derivation for application that has build with side effects a frustrating experience, but that brings reproducibility (or at least gets us really close there) so generally this is not a concern. As for splitting build into steps. I am not entirely sure what are you trying to do, but you should not need to do that. Each package in Nix has a hash, which is generated from things like hash of the source code, of the dependencies, compile flags, system architecture etc. This means that you could have multiple versions of the same package in /nix/store with different compile options or different dependencies. When you need a given dependency you know all the information to generate the hash and can easily know if you need to build it or you can use cached version. This is what makes binary cache a pretty much plug and play and don't need to worry what files to cache or whether you should split build into stages.

aidenn0 3y ago

You tell it; when you define a derivation, you are promising it is at least logically deterministic.