5 ms·

tl;dr: Nix will soon be deciding whether to rebuild packages based on the dependency contents rather than the build inputs, which should result in much fewer re

by l0b0 5y ago

tl;dr: Nix will soon be deciding whether to rebuild packages based on the dependency contents rather than the build inputs, which should result in much fewer rebuilds. This is excellent news, and hopefully the savings will end up being used to run more aggressive tests and builds of lower priority packages, to have fewer breakages.

chriswarbo 5y ago
My biggest problem using Nix has been minor changes, like the address of upstream sources, rippling down through dependencies. Hopefully this will improve with content addressing, and make it easier to fetching from places like IPFS. As an example: - I wrote a piece of software as part of a research project, which I kept in git and built using Nix. - That tool was generally useful, so I didn't want to pollute its repo with my particular experiment scripts/results, etc. so I kept those in a separate repo which pinned a particular commit of that tool. - The writeups/papers based on these experiments had a lot of churn (as usual), so I kept those in a third git repo, which pinned a particular commit of the experimental results. These writeups did some analysis, generated graphs, etc. which was also managed by Nix. This all worked great, everything was reproducible, identified by unforgable git commit IDs, etc. Then Microsoft bought GitHub and a bunch of people/projects deleted their repos (e.g. moving to GitLab, or wherever). That broke the tool's build, making the whole chain from tool -> experiment -> paper unreproducible. It was easy enough to fix by swapping the URLs in the .nix files, but that changed the git commit ID; hence the existing experimental data was tied to a commit which didn't work. Content-addressing would: - Allow monkey-patching a fix, by letting me declare that both commits produce the same tool (despite their sources containing slightly different .nix files), and hence this change doesn't alter the experimental results. - Allow such issues to be prevented, by making it easier to fetch dependencies from content-addressed storage like IPFS, which reduces the reliance on third-party servers keeping URLs alive; e.g. I could keep hosting the desired sources myself. This doesn't actually need content-addressed Nix builds, but that would make things easier (e.g. reducing the distinction between "source" and "build product").
- kevincox 5y ago
  For this specific example I'm confused why the output paths changed. Remote data fetches are already content addressed. That is why you need to provide the hash when you fetch them. This is still a problem in general though as small unimportant changes can cause rebuilds.
  - chriswarbo 5y ago
    Each repo defines its own Nix files; changing them (e.g. to swap github to gitlab) required new git commits; using those new git commits as our source changes the hash, since it's based on the inputs rather than the content. If it were content-based, the tool's hash wouldn't change (since the change to the .nix files in the new source commit wouldn't affect the contents of the resulting binaries).
    - kevincox 5y ago
      I meant the Nix hash which is based on the content, not the Git hash which is used for fetching.
      - chriswarbo 5y ago
        The git hashes are stored in the .nix files, which are part of the repo contents. This gives us a Merkle tree that pins all of the versions of all of the dependencies. Changing these git hashes hence requires changing the content of the repos. For example, we start with: tool-repo commit T1 with default.nix: myDependency = fetchGit { url = "https://....github.com/myDependency..."; rev = "...123..."; } experiment-repo commit E1 with default.nix: tool = fetchGit { url = "http://...tool-repo..."; rev = "...T1..."; } experiment-repo also contains results in a file like data/machine1/T1.json writeup-repo commit W1 with default.nix: data = fetchGit { url = "http://...experiment-repo..."; rev = "...E1..."; } When myDependency disappears from github, commit T1 becomes unbuildable; and hence the experimental results data/machine1/T1.json can't be reproduced. To fix this we need to change to the new gitlab URL: tool-repo commit T2 with default.nix: myDependency = fetchGit { url = "https://....gitlab.com/myDependency..."; rev = "...123..."; } This needs to propagate to experiment-repo. We need to decide how to handle the existing data: - We could put a note to users that we pinkie-promise that T1.json is reproducible by commit T2 (despite having no hashes/etc. to attest to this) - We could rename T1.json to T2.json, which is dishonest since that's not where the data came from. - We could re-run the experiment, which takes several days, doubles the amount of data stored in the repo and gives slight differences w.r.t. timing (since the experiments are mostly about the performance/scaling behaviour of the algorithms used by the tool). In any case, we then update the .nix files: experiment-repo commit E2 with default.nix: tool = fetchGit { url = "http://...tool-repo..."; rev = "...T2..."; } Using commit T2 instead of T1 will give T1's `tool` derivation a different hash to T2's `tool` derivation. Even though the git checkouts are fixed-output, the contents of T1 and T2 are different (one contains default.nix with a github URL, the other contains default.nix with a gitlab URL). Since we're building from different sources, the resulting tool will get a different input-based hash (if we were using content-based hashes, that wouldn't change). We then update the writeup-repo so that anyone trying to recreate our paper's results will not find the broken commit: writeup-repo commit W2 with default.nix: data = fetchGit { url = "http://...experiment-repo..."; rev = "...E2..."; } This will also change the input-based hash of the paper, although that's not too bad since rendering the paper (+ graphs, etc.) is very quick compared to the days required to run the experiments.

by l0b0 5y ago

chriswarbo 5y ago

My biggest problem using Nix has been minor changes, like the address of upstream sources, rippling down through dependencies. Hopefully this will improve with content addressing, and make it easier to fetching from places like IPFS. As an example: - I wrote a piece of software as part of a research project, which I kept in git and built using Nix. - That tool was generally useful, so I didn't want to pollute its repo with my particular experiment scripts/results, etc. so I kept those in a separate repo which pinned a particular commit of that tool. - The writeups/papers based on these experiments had a lot of churn (as usual), so I kept those in a third git repo, which pinned a particular commit of the experimental results. These writeups did some analysis, generated graphs, etc. which was also managed by Nix. This all worked great, everything was reproducible, identified by unforgable git commit IDs, etc. Then Microsoft bought GitHub and a bunch of people/projects deleted their repos (e.g. moving to GitLab, or wherever). That broke the tool's build, making the whole chain from tool -> experiment -> paper unreproducible. It was easy enough to fix by swapping the URLs in the .nix files, but that changed the git commit ID; hence the existing experimental data was tied to a commit which didn't work. Content-addressing would: - Allow monkey-patching a fix, by letting me declare that both commits produce the same tool (despite their sources containing slightly different .nix files), and hence this change doesn't alter the experimental results. - Allow such issues to be prevented, by making it easier to fetch dependencies from content-addressed storage like IPFS, which reduces the reliance on third-party servers keeping URLs alive; e.g. I could keep hosting the desired sources myself. This doesn't actually need content-addressed Nix builds, but that would make things easier (e.g. reducing the distinction between "source" and "build product").

kevincox 5y ago
For this specific example I'm confused why the output paths changed. Remote data fetches are already content addressed. That is why you need to provide the hash when you fetch them. This is still a problem in general though as small unimportant changes can cause rebuilds.
- chriswarbo 5y ago
  Each repo defines its own Nix files; changing them (e.g. to swap github to gitlab) required new git commits; using those new git commits as our source changes the hash, since it's based on the inputs rather than the content. If it were content-based, the tool's hash wouldn't change (since the change to the .nix files in the new source commit wouldn't affect the contents of the resulting binaries).
  - kevincox 5y ago
    I meant the Nix hash which is based on the content, not the Git hash which is used for fetching.
    - chriswarbo 5y ago
      The git hashes are stored in the .nix files, which are part of the repo contents. This gives us a Merkle tree that pins all of the versions of all of the dependencies. Changing these git hashes hence requires changing the content of the repos. For example, we start with: tool-repo commit T1 with default.nix: myDependency = fetchGit { url = "https://....github.com/myDependency..."; rev = "...123..."; } experiment-repo commit E1 with default.nix: tool = fetchGit { url = "http://...tool-repo..."; rev = "...T1..."; } experiment-repo also contains results in a file like data/machine1/T1.json writeup-repo commit W1 with default.nix: data = fetchGit { url = "http://...experiment-repo..."; rev = "...E1..."; } When myDependency disappears from github, commit T1 becomes unbuildable; and hence the experimental results data/machine1/T1.json can't be reproduced. To fix this we need to change to the new gitlab URL: tool-repo commit T2 with default.nix: myDependency = fetchGit { url = "https://....gitlab.com/myDependency..."; rev = "...123..."; } This needs to propagate to experiment-repo. We need to decide how to handle the existing data: - We could put a note to users that we pinkie-promise that T1.json is reproducible by commit T2 (despite having no hashes/etc. to attest to this) - We could rename T1.json to T2.json, which is dishonest since that's not where the data came from. - We could re-run the experiment, which takes several days, doubles the amount of data stored in the repo and gives slight differences w.r.t. timing (since the experiments are mostly about the performance/scaling behaviour of the algorithms used by the tool). In any case, we then update the .nix files: experiment-repo commit E2 with default.nix: tool = fetchGit { url = "http://...tool-repo..."; rev = "...T2..."; } Using commit T2 instead of T1 will give T1's `tool` derivation a different hash to T2's `tool` derivation. Even though the git checkouts are fixed-output, the contents of T1 and T2 are different (one contains default.nix with a github URL, the other contains default.nix with a gitlab URL). Since we're building from different sources, the resulting tool will get a different input-based hash (if we were using content-based hashes, that wouldn't change). We then update the writeup-repo so that anyone trying to recreate our paper's results will not find the broken commit: writeup-repo commit W2 with default.nix: data = fetchGit { url = "http://...experiment-repo..."; rev = "...E2..."; } This will also change the input-based hash of the paper, although that's not too bad since rendering the paper (+ graphs, etc.) is very quick compared to the days required to run the experiments.