5 ms·

Each repo defines its own Nix files; changing them (e.g. to swap github to gitlab) required new git commits; using those new git commits as our source changes t

Each repo defines its own Nix files; changing them (e.g. to swap github to gitlab) required new git commits; using those new git commits as our source changes the hash, since it's based on the inputs rather than the content. If it were content-based, the tool's hash wouldn't change (since the change to the .nix files in the new source commit wouldn't affect the contents of the resulting binaries).

kevincox 5y ago
I meant the Nix hash which is based on the content, not the Git hash which is used for fetching.
- chriswarbo 5y ago
  The git hashes are stored in the .nix files, which are part of the repo contents. This gives us a Merkle tree that pins all of the versions of all of the dependencies. Changing these git hashes hence requires changing the content of the repos. For example, we start with: tool-repo commit T1 with default.nix: myDependency = fetchGit { url = "https://....github.com/myDependency..."; rev = "...123..."; } experiment-repo commit E1 with default.nix: tool = fetchGit { url = "http://...tool-repo..."; rev = "...T1..."; } experiment-repo also contains results in a file like data/machine1/T1.json writeup-repo commit W1 with default.nix: data = fetchGit { url = "http://...experiment-repo..."; rev = "...E1..."; } When myDependency disappears from github, commit T1 becomes unbuildable; and hence the experimental results data/machine1/T1.json can't be reproduced. To fix this we need to change to the new gitlab URL: tool-repo commit T2 with default.nix: myDependency = fetchGit { url = "https://....gitlab.com/myDependency..."; rev = "...123..."; } This needs to propagate to experiment-repo. We need to decide how to handle the existing data: - We could put a note to users that we pinkie-promise that T1.json is reproducible by commit T2 (despite having no hashes/etc. to attest to this) - We could rename T1.json to T2.json, which is dishonest since that's not where the data came from. - We could re-run the experiment, which takes several days, doubles the amount of data stored in the repo and gives slight differences w.r.t. timing (since the experiments are mostly about the performance/scaling behaviour of the algorithms used by the tool). In any case, we then update the .nix files: experiment-repo commit E2 with default.nix: tool = fetchGit { url = "http://...tool-repo..."; rev = "...T2..."; } Using commit T2 instead of T1 will give T1's `tool` derivation a different hash to T2's `tool` derivation. Even though the git checkouts are fixed-output, the contents of T1 and T2 are different (one contains default.nix with a github URL, the other contains default.nix with a gitlab URL). Since we're building from different sources, the resulting tool will get a different input-based hash (if we were using content-based hashes, that wouldn't change). We then update the writeup-repo so that anyone trying to recreate our paper's results will not find the broken commit: writeup-repo commit W2 with default.nix: data = fetchGit { url = "http://...experiment-repo..."; rev = "...E2..."; } This will also change the input-based hash of the paper, although that's not too bad since rendering the paper (+ graphs, etc.) is very quick compared to the days required to run the experiments.

kevincox 5y ago

I meant the Nix hash which is based on the content, not the Git hash which is used for fetching.

chriswarbo 5y ago
The git hashes are stored in the .nix files, which are part of the repo contents. This gives us a Merkle tree that pins all of the versions of all of the dependencies. Changing these git hashes hence requires changing the content of the repos. For example, we start with: tool-repo commit T1 with default.nix: myDependency = fetchGit { url = "https://....github.com/myDependency..."; rev = "...123..."; } experiment-repo commit E1 with default.nix: tool = fetchGit { url = "http://...tool-repo..."; rev = "...T1..."; } experiment-repo also contains results in a file like data/machine1/T1.json writeup-repo commit W1 with default.nix: data = fetchGit { url = "http://...experiment-repo..."; rev = "...E1..."; } When myDependency disappears from github, commit T1 becomes unbuildable; and hence the experimental results data/machine1/T1.json can't be reproduced. To fix this we need to change to the new gitlab URL: tool-repo commit T2 with default.nix: myDependency = fetchGit { url = "https://....gitlab.com/myDependency..."; rev = "...123..."; } This needs to propagate to experiment-repo. We need to decide how to handle the existing data: - We could put a note to users that we pinkie-promise that T1.json is reproducible by commit T2 (despite having no hashes/etc. to attest to this) - We could rename T1.json to T2.json, which is dishonest since that's not where the data came from. - We could re-run the experiment, which takes several days, doubles the amount of data stored in the repo and gives slight differences w.r.t. timing (since the experiments are mostly about the performance/scaling behaviour of the algorithms used by the tool). In any case, we then update the .nix files: experiment-repo commit E2 with default.nix: tool = fetchGit { url = "http://...tool-repo..."; rev = "...T2..."; } Using commit T2 instead of T1 will give T1's `tool` derivation a different hash to T2's `tool` derivation. Even though the git checkouts are fixed-output, the contents of T1 and T2 are different (one contains default.nix with a github URL, the other contains default.nix with a gitlab URL). Since we're building from different sources, the resulting tool will get a different input-based hash (if we were using content-based hashes, that wouldn't change). We then update the writeup-repo so that anyone trying to recreate our paper's results will not find the broken commit: writeup-repo commit W2 with default.nix: data = fetchGit { url = "http://...experiment-repo..."; rev = "...E2..."; } This will also change the input-based hash of the paper, although that's not too bad since rendering the paper (+ graphs, etc.) is very quick compared to the days required to run the experiments.