4 ms·
I think this is technically an option, but IIUC it'll have plenty of its own hurdles. The cache is populated by the results of builds run by NixOS's hydra build
by abathur 3y ago
I think this is technically an option, but IIUC it'll have plenty of its own hurdles. The cache is populated by the results of builds run by NixOS's hydra build farm over a period of many years, and hydra stays busy enough that it's hard to imagine this approach being viable without standing up a separate cluster of build machines dedicated to the task. To complete it in a meaningful timeframe, the cluster might also need to be quite a bit bigger.
(NixOS hydra obviously has some spare capacity or it'd never catch up, but I would be surprised if it has enough idle capacity to rebuild everything over any meaningful timeframe without also significantly affecting velocity on everything else.)
Calling on the community to dedicate temporary build resources might work--but the systems need to be trusted, so it can't quite be Folding@home...
I guess if there's a bit of Pareto in the distribution of build output sizes, there might be meaningful savings in rebuilding the packages that take up the most storage and paying to move the small stuff. It might also be possible to triage the sources in the cache--re-fetch all that are still available from the original source or a mirror, and then just export the ones that either couldn't be fetched or could be fetched but the contents no longer match the hash.
- JonChesterfield 3y agoProbably does take a lot of compute to recreate the cache, but doing it in order of requests works pretty well as a heuristic for most important first, gradually backfilling when idle. Adding machines to the build infra seems like less of a loss than paying egress fees too. Overall seems plausible that this data store isn't just a cache of things that can be recreated at will. The sibling comment about source being unavailable (or still available but at a different location) for the earlier builds is interesting in that respect.
- RaitoBezarius 3y agoIt is also probably impossible given some sources completely disappeared.