3 ms·

While I can't speak for the person you're replying to, the technology at least exists. Binary diffs are sometimes used to distribute game updates, where you're

While I can't speak for the person you're replying to, the technology at least exists. Binary diffs are sometimes used to distribute game updates, where you're saving on bandwidth for thousands if not millions of players - which costs enough $$$ to actually be worth optimizing for. On the other hand, between simpler designs and content encryption being sometimes at odds with content compression... so is just sending the full 10MB. For a VCS - I'd probably be happy enough to just have storage compression - using any of the standard tools on the combination. > You don't really want to record a full 10MB of data every time you change one pixel in your texture or one blip in your sound, right? Actual changes to content in a gamedev studio are very unlikely to be as small as a single pixel. Changes to source code are unlikely to be as small as a single character either. And we definitely want a record of that 10MB. We're willing to sacrifice some of our CI build history. Maybe only keeping ~weekly archives, or milestone/QAed builds after awhile, of dozens or hundreds of GB - and maybe eventually getting rid of some of the really old ones eventually. Having an exact binary copy of a build a bug was reported against can be incredibly useful.

tyingq 7y ago
I usually see bsdiff or courgette cited as good tools for binary diffs: http://www.daemonology.net/bsdiff/ http://www.daemonology.net/bsdiff/ https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...
chrisweekly 7y ago
"Having an exact binary copy of a build a bug was reported against can be incredibly useful." Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern?
- MaulingMonkey 7y ago
  > Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern? One person's immutable build artifact is another person's vendored build input. It's common to vendor third party libraries by uploading their immutable build artifacts (.dll, .so, .a, .lib, etc.) into your VCS, handling distribution, and keeping track of which versions were used for any given build. It makes a lot of sense if those third party libraries are slow to build, rarely modified, and/or closed source - no sense wasting dev time forcing them to rebuild it all from scratch. The next logical step is to have a build server auto-upload said immutable build artifacts into your VCS, for those third party libraries that you do have source code for, when your VCS copy of said source is modified. Much more secure and reproducable than having random devs do it. And hey, if your build servers are already uploading build artifacts to VCS for third party libraries, why not do so for your own first party build artifacts too? Tools devs spending most of their time in C# probably don't need to spend hours rebuilding the accompanying C++ engine it interoperates with from scratch, for example, so why not "vendor" the engine to improve their iteration times? This can lead to dozens of gigs of mostly identical immutable build artifacts reuploaded into your VCS several times per day, with QA testing and then integrating those build artifacts into other branches on top of that. The occasional 10MB png is no longer noticable by comparison.
  - chrisweekly 7y ago
    Yeah, I get it. Still seems a stretch to fault git for "failing" to optimize for that inefficient-by-design use case though.
    - MaulingMonkey 7y ago
      To be fair, I mostly don't fault git for failing to optimize that far, even if there are alternatives that do. That's far enough outside the core use case for myself and those I know that I'd be willing to sacrifice it for other, more important considerations. But I'm totally willing to fault git for failing to optimize enough to handle the manual commit cadence of source game assets though. Because that's not just a tertiary use case - frequently for coworkers it's their primary use case. The end result is I mostly only use git for personal hobby stuff, where it's a secondary use case and my assets are sufficiently small as to not cause problems.
      - chrisweekly 7y ago
        Right on. Thanks for clarifying -- and for confirming a legitimate complaint based on real-world, personal experience.
  - Nullabillity 7y ago
    I can sympathize with the game assets argument, but this problem is just the result of trying to stuff a square peg into the round hole. Build artifact caching is a different problem from source control, with very different requirements: 1. As you mentioned, the artifacts tend to get huge. 2. The cache needs to be easy to bypass. From your example, it needs to be easy for the C++ engine devs to do builds like "the game but with the new engine" to test out their changes. 3. The cache needs to be precise, so you don't end up with mystery errors once it finally does trigger, or people wondering why their changes don't seem to apply. 4. The builds need to be exactly reproducible, so you don't end up with some critical package that only Steve Who Left 5 Years Ago could build (or Jenkins Node 3 That Just Suffered A Critical HDD Failure). Git either doesn't care about or fails spectacularly for each of those points. In particular, #3 will be very confusing since there will be a delay between the code push and the related build push. Nix[0] solves #2 and #3 by caching build artifacts (both locally and remotely[1][2][3]) based on code hashes and a dependency DAG (for each subproject or build artifact, so changing subproject X won't trigger a rebuild of unrelated subproject Y, but will rebuild Z that depends on X). It helps with #4 by performing all builds in an isolated sandbox. #1 is solved by evicting old artifacts, which is safe as long as you trust #4. If the old artifact is needed again then it will be rebuilt for you transparently. Currently this is done by evicting the oldest artifacts first, but it could be an interesting project to add a cost/benefit bias here (how long did it take to build this artifact, vs the amount of space it consumes?). [0]: https://builtwithnix.org/ https://builtwithnix.org/ [1]: https://nixos.wiki/wiki/Binary_Cache https://nixos.wiki/wiki/Binary_Cache [2]: https://nixos.org/nix/manual/#sec-sharing-packages https://nixos.org/nix/manual/#sec-sharing-packages [3]: https://cachix.org/ https://cachix.org/

tyingq 7y ago

I usually see bsdiff or courgette cited as good tools for binary diffs: http://www.daemonology.net/bsdiff/ http://www.daemonology.net/bsdiff/ https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...

chrisweekly 7y ago

"Having an exact binary copy of a build a bug was reported against can be incredibly useful." Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern?

MaulingMonkey 7y ago
> Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern? One person's immutable build artifact is another person's vendored build input. It's common to vendor third party libraries by uploading their immutable build artifacts (.dll, .so, .a, .lib, etc.) into your VCS, handling distribution, and keeping track of which versions were used for any given build. It makes a lot of sense if those third party libraries are slow to build, rarely modified, and/or closed source - no sense wasting dev time forcing them to rebuild it all from scratch. The next logical step is to have a build server auto-upload said immutable build artifacts into your VCS, for those third party libraries that you do have source code for, when your VCS copy of said source is modified. Much more secure and reproducable than having random devs do it. And hey, if your build servers are already uploading build artifacts to VCS for third party libraries, why not do so for your own first party build artifacts too? Tools devs spending most of their time in C# probably don't need to spend hours rebuilding the accompanying C++ engine it interoperates with from scratch, for example, so why not "vendor" the engine to improve their iteration times? This can lead to dozens of gigs of mostly identical immutable build artifacts reuploaded into your VCS several times per day, with QA testing and then integrating those build artifacts into other branches on top of that. The occasional 10MB png is no longer noticable by comparison.
- chrisweekly 7y ago
  Yeah, I get it. Still seems a stretch to fault git for "failing" to optimize for that inefficient-by-design use case though.
  - MaulingMonkey 7y ago
    To be fair, I mostly don't fault git for failing to optimize that far, even if there are alternatives that do. That's far enough outside the core use case for myself and those I know that I'd be willing to sacrifice it for other, more important considerations. But I'm totally willing to fault git for failing to optimize enough to handle the manual commit cadence of source game assets though. Because that's not just a tertiary use case - frequently for coworkers it's their primary use case. The end result is I mostly only use git for personal hobby stuff, where it's a secondary use case and my assets are sufficiently small as to not cause problems.
    - chrisweekly 7y ago
      Right on. Thanks for clarifying -- and for confirming a legitimate complaint based on real-world, personal experience.
- Nullabillity 7y ago
  I can sympathize with the game assets argument, but this problem is just the result of trying to stuff a square peg into the round hole. Build artifact caching is a different problem from source control, with very different requirements: 1. As you mentioned, the artifacts tend to get huge. 2. The cache needs to be easy to bypass. From your example, it needs to be easy for the C++ engine devs to do builds like "the game but with the new engine" to test out their changes. 3. The cache needs to be precise, so you don't end up with mystery errors once it finally does trigger, or people wondering why their changes don't seem to apply. 4. The builds need to be exactly reproducible, so you don't end up with some critical package that only Steve Who Left 5 Years Ago could build (or Jenkins Node 3 That Just Suffered A Critical HDD Failure). Git either doesn't care about or fails spectacularly for each of those points. In particular, #3 will be very confusing since there will be a delay between the code push and the related build push. Nix[0] solves #2 and #3 by caching build artifacts (both locally and remotely[1][2][3]) based on code hashes and a dependency DAG (for each subproject or build artifact, so changing subproject X won't trigger a rebuild of unrelated subproject Y, but will rebuild Z that depends on X). It helps with #4 by performing all builds in an isolated sandbox. #1 is solved by evicting old artifacts, which is safe as long as you trust #4. If the old artifact is needed again then it will be rebuilt for you transparently. Currently this is done by evicting the oldest artifacts first, but it could be an interesting project to add a cost/benefit bias here (how long did it take to build this artifact, vs the amount of space it consumes?). [0]: https://builtwithnix.org/ https://builtwithnix.org/ [1]: https://nixos.wiki/wiki/Binary_Cache https://nixos.wiki/wiki/Binary_Cache [2]: https://nixos.org/nix/manual/#sec-sharing-packages https://nixos.org/nix/manual/#sec-sharing-packages [3]: https://cachix.org/ https://cachix.org/