5 ms·

> Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern? One person's immutable build artifact is another person's ve

> Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern? One person's immutable build artifact is another person's vendored build input. It's common to vendor third party libraries by uploading their immutable build artifacts (.dll, .so, .a, .lib, etc.) into your VCS, handling distribution, and keeping track of which versions were used for any given build. It makes a lot of sense if those third party libraries are slow to build, rarely modified, and/or closed source - no sense wasting dev time forcing them to rebuild it all from scratch. The next logical step is to have a build server auto-upload said immutable build artifacts into your VCS, for those third party libraries that you do have source code for, when your VCS copy of said source is modified. Much more secure and reproducable than having random devs do it. And hey, if your build servers are already uploading build artifacts to VCS for third party libraries, why not do so for your own first party build artifacts too? Tools devs spending most of their time in C# probably don't need to spend hours rebuilding the accompanying C++ engine it interoperates with from scratch, for example, so why not "vendor" the engine to improve their iteration times? This can lead to dozens of gigs of mostly identical immutable build artifacts reuploaded into your VCS several times per day, with QA testing and then integrating those build artifacts into other branches on top of that. The occasional 10MB png is no longer noticable by comparison.

chrisweekly 7y ago
Yeah, I get it. Still seems a stretch to fault git for "failing" to optimize for that inefficient-by-design use case though.
- MaulingMonkey 7y ago
  To be fair, I mostly don't fault git for failing to optimize that far, even if there are alternatives that do. That's far enough outside the core use case for myself and those I know that I'd be willing to sacrifice it for other, more important considerations. But I'm totally willing to fault git for failing to optimize enough to handle the manual commit cadence of source game assets though. Because that's not just a tertiary use case - frequently for coworkers it's their primary use case. The end result is I mostly only use git for personal hobby stuff, where it's a secondary use case and my assets are sufficiently small as to not cause problems.
  - chrisweekly 7y ago
    Right on. Thanks for clarifying -- and for confirming a legitimate complaint based on real-world, personal experience.
Nullabillity 7y ago
I can sympathize with the game assets argument, but this problem is just the result of trying to stuff a square peg into the round hole. Build artifact caching is a different problem from source control, with very different requirements: 1. As you mentioned, the artifacts tend to get huge. 2. The cache needs to be easy to bypass. From your example, it needs to be easy for the C++ engine devs to do builds like "the game but with the new engine" to test out their changes. 3. The cache needs to be precise, so you don't end up with mystery errors once it finally does trigger, or people wondering why their changes don't seem to apply. 4. The builds need to be exactly reproducible, so you don't end up with some critical package that only Steve Who Left 5 Years Ago could build (or Jenkins Node 3 That Just Suffered A Critical HDD Failure). Git either doesn't care about or fails spectacularly for each of those points. In particular, #3 will be very confusing since there will be a delay between the code push and the related build push. Nix[0] solves #2 and #3 by caching build artifacts (both locally and remotely[1][2][3]) based on code hashes and a dependency DAG (for each subproject or build artifact, so changing subproject X won't trigger a rebuild of unrelated subproject Y, but will rebuild Z that depends on X). It helps with #4 by performing all builds in an isolated sandbox. #1 is solved by evicting old artifacts, which is safe as long as you trust #4. If the old artifact is needed again then it will be rebuilt for you transparently. Currently this is done by evicting the oldest artifacts first, but it could be an interesting project to add a cost/benefit bias here (how long did it take to build this artifact, vs the amount of space it consumes?). [0]: https://builtwithnix.org/ https://builtwithnix.org/ [1]: https://nixos.wiki/wiki/Binary_Cache https://nixos.wiki/wiki/Binary_Cache [2]: https://nixos.org/nix/manual/#sec-sharing-packages https://nixos.org/nix/manual/#sec-sharing-packages [3]: https://cachix.org/ https://cachix.org/
- MaulingMonkey 7y ago
  Assets and code have mostly the same needs out of a version control system - diffs, history, control over versions, etc. - and there are version control systems which handle both adequately. That said, I'll grant git is quite focused on code version control specifically - and I would not dream of trying to scale assets into it directly. > 1. As you mentioned, the artifacts tend to get huge. This, admittedly, is more common with build artifacts. That said, I've hit quota limits with autogenerated binding code on crates.io, with several hundred megs of code still being in the double digits when better compressed by cargo than I can figure out how to compress with 7-zip. And that's a small single person hobby project, not a google monorepository. > 2. The cache needs to be easy to bypass I need to bypass locally vendored source code frequently as well, to test upstream patches etc. > 3. The cache needs to be precise, so you don't end up with mystery errors once it finally does trigger, or people wondering why their changes don't seem to apply. Also entirely true of source code. > 4. The builds need to be exactly reproducible, so you don't end up with some critical package that only Steve Who Left 5 Years Ago could build (or Jenkins Node 3 That Just Suffered A Critical HDD Failure). Enshrining built libs in VCS is an alternative tackling of the problem. You might not be able to reproduce that exact build bit-for-bit thanks to who knows what minor compiler updates have been forced upon you, but at least you'll have the immutable original to reproduce bugs against. > In particular, #3 will be very confusing since there will be a delay between the code push and the related build push. It's already extremely common - in the name of build stability, including with git - to protect a branch from direct push, and have CI generate and delay committing a merge until it's verified the build goes green. By wonderful coincidence, this is also well after CI has finished building those artifacts - in fact, it's been running tests against those artifacts - so it can atomically commit the source merge + binaries of said source merge all at once. No delay between the two. There are some caveats - gathering the binaries can be a pain for some CI systems, or perhaps your build farm is underfunded and can only reasonably build a subset of your build matrix before merging. Or perhaps the person setting it up didn't think it through and has set things up such that code reaches a branch that uses VCS libs before the built libs reach the same spot in VCS - I'll admit I've experienced that, and it's horrible. Nix, Incredibuild, etc. are wonderful alternatives to tackle the problem from a different angle though.

chrisweekly 7y ago

Yeah, I get it. Still seems a stretch to fault git for "failing" to optimize for that inefficient-by-design use case though.

MaulingMonkey 7y ago
To be fair, I mostly don't fault git for failing to optimize that far, even if there are alternatives that do. That's far enough outside the core use case for myself and those I know that I'd be willing to sacrifice it for other, more important considerations. But I'm totally willing to fault git for failing to optimize enough to handle the manual commit cadence of source game assets though. Because that's not just a tertiary use case - frequently for coworkers it's their primary use case. The end result is I mostly only use git for personal hobby stuff, where it's a secondary use case and my assets are sufficiently small as to not cause problems.
- chrisweekly 7y ago
  Right on. Thanks for clarifying -- and for confirming a legitimate complaint based on real-world, personal experience.

Nullabillity 7y ago

I can sympathize with the game assets argument, but this problem is just the result of trying to stuff a square peg into the round hole. Build artifact caching is a different problem from source control, with very different requirements: 1. As you mentioned, the artifacts tend to get huge. 2. The cache needs to be easy to bypass. From your example, it needs to be easy for the C++ engine devs to do builds like "the game but with the new engine" to test out their changes. 3. The cache needs to be precise, so you don't end up with mystery errors once it finally does trigger, or people wondering why their changes don't seem to apply. 4. The builds need to be exactly reproducible, so you don't end up with some critical package that only Steve Who Left 5 Years Ago could build (or Jenkins Node 3 That Just Suffered A Critical HDD Failure). Git either doesn't care about or fails spectacularly for each of those points. In particular, #3 will be very confusing since there will be a delay between the code push and the related build push. Nix[0] solves #2 and #3 by caching build artifacts (both locally and remotely[1][2][3]) based on code hashes and a dependency DAG (for each subproject or build artifact, so changing subproject X won't trigger a rebuild of unrelated subproject Y, but will rebuild Z that depends on X). It helps with #4 by performing all builds in an isolated sandbox. #1 is solved by evicting old artifacts, which is safe as long as you trust #4. If the old artifact is needed again then it will be rebuilt for you transparently. Currently this is done by evicting the oldest artifacts first, but it could be an interesting project to add a cost/benefit bias here (how long did it take to build this artifact, vs the amount of space it consumes?). [0]: https://builtwithnix.org/ https://builtwithnix.org/ [1]: https://nixos.wiki/wiki/Binary_Cache https://nixos.wiki/wiki/Binary_Cache [2]: https://nixos.org/nix/manual/#sec-sharing-packages https://nixos.org/nix/manual/#sec-sharing-packages [3]: https://cachix.org/ https://cachix.org/

MaulingMonkey 7y ago
Assets and code have mostly the same needs out of a version control system - diffs, history, control over versions, etc. - and there are version control systems which handle both adequately. That said, I'll grant git is quite focused on code version control specifically - and I would not dream of trying to scale assets into it directly. > 1. As you mentioned, the artifacts tend to get huge. This, admittedly, is more common with build artifacts. That said, I've hit quota limits with autogenerated binding code on crates.io, with several hundred megs of code still being in the double digits when better compressed by cargo than I can figure out how to compress with 7-zip. And that's a small single person hobby project, not a google monorepository. > 2. The cache needs to be easy to bypass I need to bypass locally vendored source code frequently as well, to test upstream patches etc. > 3. The cache needs to be precise, so you don't end up with mystery errors once it finally does trigger, or people wondering why their changes don't seem to apply. Also entirely true of source code. > 4. The builds need to be exactly reproducible, so you don't end up with some critical package that only Steve Who Left 5 Years Ago could build (or Jenkins Node 3 That Just Suffered A Critical HDD Failure). Enshrining built libs in VCS is an alternative tackling of the problem. You might not be able to reproduce that exact build bit-for-bit thanks to who knows what minor compiler updates have been forced upon you, but at least you'll have the immutable original to reproduce bugs against. > In particular, #3 will be very confusing since there will be a delay between the code push and the related build push. It's already extremely common - in the name of build stability, including with git - to protect a branch from direct push, and have CI generate and delay committing a merge until it's verified the build goes green. By wonderful coincidence, this is also well after CI has finished building those artifacts - in fact, it's been running tests against those artifacts - so it can atomically commit the source merge + binaries of said source merge all at once. No delay between the two. There are some caveats - gathering the binaries can be a pain for some CI systems, or perhaps your build farm is underfunded and can only reasonably build a subset of your build matrix before merging. Or perhaps the person setting it up didn't think it through and has set things up such that code reaches a branch that uses VCS libs before the built libs reach the same spot in VCS - I'll admit I've experienced that, and it's horrible. Nix, Incredibuild, etc. are wonderful alternatives to tackle the problem from a different angle though.