2 ms·
Something does not compute... problem: > Have you ever had a build fail because of a network error on some third-party server? Commit your dependencies and tha
by theamk 3y ago
Something does not compute... problem:
> Have you ever had a build fail because of a network error on some third-party server? Commit your dependencies and that will never happen.
author's "ideal" solution:
> First, the user clones a random GenAI repo. This is near instantaneous as files are not prefetched. The user then invokes the build script. As files are accessed they're downloaded. The very first build may download a few hundred megabytes of data.
Seems like contradiction to me? Do you want your builds to be offline or online?
I have a feeling that the real problem is that (1) author never worked with build systems which supply their own compiler, like buildroot or hermetic bazel and (2) author only downloaded dependencies from third-party servers.
You can solve all the problems in the blog post by (1) changing build system to download compilers and libraries and (2) adopting some sort of "binary storage" (something as simple as private ftp server + small text file checked into repo with path and checksum + few scripts to upload and download). And it would not require any new major code investment, nor new tooling, nor new VCS.
The key idea is that all dependencies should always come from your local binary server, never from the third-party server. This is a bit of PITA as many modern tools default to downloading stuff from third-party servers, and it could be hard to convince them to look at your local server instead. But once you do it, you get reliability + reproducibility + sustainability, without having to reinvent the world.