5 ms·

I work for a 40-people game studio. A major limitation of git is how it deals with many "big" (~10Mb) binary files (3D models, textures, sounds, etc.). We end

by Ace17 7y ago

I work for a 40-people game studio. A major limitation of git is how it deals with many "big" (~10Mb) binary files (3D models, textures, sounds, etc.). We ended up developing our own layer over git, and we're very happy ; even git-lfs can't provide similar benefits. This technique seems to be commonplace for game studios (e.g Naughty Dog, Bungee), so certainly git has room for improvement here.

onion2k 7y ago
Has any studio open sourced what they've built? Or turned it in to a product? It seems there could be an opportunity to do something before git solves the problems for that use case.
rustybolt 7y ago
Jup, I experienced this too (not at a game studio though, and the team I worked with wasn't nearly experienced enough to write a layer over git). When we switched to a new version of the git gui it would stop working because when you click through the GUI, it would perform some git operations that were supposed to run fast. I filed an issue that quickly got shot down with 'wontfix, your repo is too large and git is not for binary files'.
taftster 7y ago
Do you have tools that you can utilize diffs from your binary file changes? Or does a change simply just replace all the bytes. I'd argue if it's the later, that git was never the right choice to begin with. You don't really want to record a full 10MB of data every time you change one pixel in your texture or one blip in your sound, right? So I don't know if this is a "major limitation" of git per se. Not saying there's a better solution off-the-shelf (you're obviously happy with your home grown). But this was probably never a realistic use for git in the first place.
- MaulingMonkey 7y ago
  While I can't speak for the person you're replying to, the technology at least exists. Binary diffs are sometimes used to distribute game updates, where you're saving on bandwidth for thousands if not millions of players - which costs enough $$$ to actually be worth optimizing for. On the other hand, between simpler designs and content encryption being sometimes at odds with content compression... so is just sending the full 10MB. For a VCS - I'd probably be happy enough to just have storage compression - using any of the standard tools on the combination. > You don't really want to record a full 10MB of data every time you change one pixel in your texture or one blip in your sound, right? Actual changes to content in a gamedev studio are very unlikely to be as small as a single pixel. Changes to source code are unlikely to be as small as a single character either. And we definitely want a record of that 10MB. We're willing to sacrifice some of our CI build history. Maybe only keeping ~weekly archives, or milestone/QAed builds after awhile, of dozens or hundreds of GB - and maybe eventually getting rid of some of the really old ones eventually. Having an exact binary copy of a build a bug was reported against can be incredibly useful.
  - tyingq 7y ago
    I usually see bsdiff or courgette cited as good tools for binary diffs: http://www.daemonology.net/bsdiff/ http://www.daemonology.net/bsdiff/ https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...
  - chrisweekly 7y ago
    "Having an exact binary copy of a build a bug was reported against can be incredibly useful." Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern?
    - MaulingMonkey 7y ago
      > Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern? One person's immutable build artifact is another person's vendored build input. It's common to vendor third party libraries by uploading their immutable build artifacts (.dll, .so, .a, .lib, etc.) into your VCS, handling distribution, and keeping track of which versions were used for any given build. It makes a lot of sense if those third party libraries are slow to build, rarely modified, and/or closed source - no sense wasting dev time forcing them to rebuild it all from scratch. The next logical step is to have a build server auto-upload said immutable build artifacts into your VCS, for those third party libraries that you do have source code for, when your VCS copy of said source is modified. Much more secure and reproducable than having random devs do it. And hey, if your build servers are already uploading build artifacts to VCS for third party libraries, why not do so for your own first party build artifacts too? Tools devs spending most of their time in C# probably don't need to spend hours rebuilding the accompanying C++ engine it interoperates with from scratch, for example, so why not "vendor" the engine to improve their iteration times? This can lead to dozens of gigs of mostly identical immutable build artifacts reuploaded into your VCS several times per day, with QA testing and then integrating those build artifacts into other branches on top of that. The occasional 10MB png is no longer noticable by comparison.
      - chrisweekly 7y ago
- smcameron 7y ago
  > You don't really want to record a full 10MB of data every time you change one pixel in your texture or one blip in your sound, right? Ideally, yes, why wouldn't I? I want to capture the exact state of the thing at each change.
  - taftster 7y ago
    I kind of phrased that poorly. I should have added the context of "in git". Saving a new 10MB file every time you change it, as per my original premise, is not something that git was really designed for. It's asking a screwdriver to do the work of a hammer. I totally get the use case of saving each iteration of that 10MB file _somewhere_. But expecting git to do that job is not the right level of expectation, was my main point. When I have worked with binaries like that described, I will place a URI reference to a file hash and have something that knows how to resolve it. A file store (think S3 or whatever) that has files named: texture1.dat-[sha1] and change the reference to the file in the source. e.g. a "poor man's" version control by way of file naming conventions. Does this approach work in your world?
mathw 7y ago
This does not surprise me. Git's original purpose of managing versions of a tree of text files (i.e. the source code of the Linux kernel) pervasively influences it, and I wouldn't expect it to be any good for working with binary files or large files. If somebody comes up with something that matches Git's strengths and also handles binaries and biggies much, much better then they could definitely topple Git with it. It'd take time for the word to spread, the tools to mature and the hosting to appear, but I can definitely see it happening. I think most people know that Git isn't perfect, but it's also the case that coming up with anything better is an extremely difficult task. If it wasn't, someone would have already done it. It's not like people haven't been trying.
AndriyKunitsyn 7y ago
The usual technique of game studios is using Perforce. It is clunky and sometimes straight up infuriating, but it handles large files well.
zeofig 7y ago
Yeah, git with binary files is not fun. Have you tried git annex?
j88439h84 7y ago
What does your layer do?
erikbye 7y ago
Why did you not use Plastic SCM or Perforce?
- 1auralynn 7y ago
  I can't recommend Plastic enough - it's so fast and a great UI but super powerful. I've been using it at my company for years.
DaiPlusPlus 7y ago
Aren’t game studios and other creative studios meant to use “asset management” systems instead for their large binaries? Diffing a PSD as a binary is impossible - whereas proper asset management tools will deconstruct the PSD’s format to make for a human-readable diff (e.g. added/removed layers, properties, etc).
- lodi 7y ago
  Separate version control for code vs assets leads to a world of pain. Also you can use whatever diff tool you want; doesn't have to be the built-in textual diff.

by Ace17 7y ago

onion2k 7y ago

Has any studio open sourced what they've built? Or turned it in to a product? It seems there could be an opportunity to do something before git solves the problems for that use case.

rustybolt 7y ago

Jup, I experienced this too (not at a game studio though, and the team I worked with wasn't nearly experienced enough to write a layer over git). When we switched to a new version of the git gui it would stop working because when you click through the GUI, it would perform some git operations that were supposed to run fast. I filed an issue that quickly got shot down with 'wontfix, your repo is too large and git is not for binary files'.

taftster 7y ago

Do you have tools that you can utilize diffs from your binary file changes? Or does a change simply just replace all the bytes. I'd argue if it's the later, that git was never the right choice to begin with. You don't really want to record a full 10MB of data every time you change one pixel in your texture or one blip in your sound, right? So I don't know if this is a "major limitation" of git per se. Not saying there's a better solution off-the-shelf (you're obviously happy with your home grown). But this was probably never a realistic use for git in the first place.

MaulingMonkey 7y ago
While I can't speak for the person you're replying to, the technology at least exists. Binary diffs are sometimes used to distribute game updates, where you're saving on bandwidth for thousands if not millions of players - which costs enough $$$ to actually be worth optimizing for. On the other hand, between simpler designs and content encryption being sometimes at odds with content compression... so is just sending the full 10MB. For a VCS - I'd probably be happy enough to just have storage compression - using any of the standard tools on the combination. > You don't really want to record a full 10MB of data every time you change one pixel in your texture or one blip in your sound, right? Actual changes to content in a gamedev studio are very unlikely to be as small as a single pixel. Changes to source code are unlikely to be as small as a single character either. And we definitely want a record of that 10MB. We're willing to sacrifice some of our CI build history. Maybe only keeping ~weekly archives, or milestone/QAed builds after awhile, of dozens or hundreds of GB - and maybe eventually getting rid of some of the really old ones eventually. Having an exact binary copy of a build a bug was reported against can be incredibly useful.
- tyingq 7y ago
  I usually see bsdiff or courgette cited as good tools for binary diffs: http://www.daemonology.net/bsdiff/ http://www.daemonology.net/bsdiff/ https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...
- chrisweekly 7y ago
  "Having an exact binary copy of a build a bug was reported against can be incredibly useful." Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern?
  - MaulingMonkey 7y ago
    > Sure, immutable build artifacts can be invaluable -- but aren't they also an orthogonal concern? One person's immutable build artifact is another person's vendored build input. It's common to vendor third party libraries by uploading their immutable build artifacts (.dll, .so, .a, .lib, etc.) into your VCS, handling distribution, and keeping track of which versions were used for any given build. It makes a lot of sense if those third party libraries are slow to build, rarely modified, and/or closed source - no sense wasting dev time forcing them to rebuild it all from scratch. The next logical step is to have a build server auto-upload said immutable build artifacts into your VCS, for those third party libraries that you do have source code for, when your VCS copy of said source is modified. Much more secure and reproducable than having random devs do it. And hey, if your build servers are already uploading build artifacts to VCS for third party libraries, why not do so for your own first party build artifacts too? Tools devs spending most of their time in C# probably don't need to spend hours rebuilding the accompanying C++ engine it interoperates with from scratch, for example, so why not "vendor" the engine to improve their iteration times? This can lead to dozens of gigs of mostly identical immutable build artifacts reuploaded into your VCS several times per day, with QA testing and then integrating those build artifacts into other branches on top of that. The occasional 10MB png is no longer noticable by comparison.
    - chrisweekly 7y ago
smcameron 7y ago
> You don't really want to record a full 10MB of data every time you change one pixel in your texture or one blip in your sound, right? Ideally, yes, why wouldn't I? I want to capture the exact state of the thing at each change.
- taftster 7y ago
  I kind of phrased that poorly. I should have added the context of "in git". Saving a new 10MB file every time you change it, as per my original premise, is not something that git was really designed for. It's asking a screwdriver to do the work of a hammer. I totally get the use case of saving each iteration of that 10MB file _somewhere_. But expecting git to do that job is not the right level of expectation, was my main point. When I have worked with binaries like that described, I will place a URI reference to a file hash and have something that knows how to resolve it. A file store (think S3 or whatever) that has files named: texture1.dat-[sha1] and change the reference to the file in the source. e.g. a "poor man's" version control by way of file naming conventions. Does this approach work in your world?

mathw 7y ago

This does not surprise me. Git's original purpose of managing versions of a tree of text files (i.e. the source code of the Linux kernel) pervasively influences it, and I wouldn't expect it to be any good for working with binary files or large files. If somebody comes up with something that matches Git's strengths and also handles binaries and biggies much, much better then they could definitely topple Git with it. It'd take time for the word to spread, the tools to mature and the hosting to appear, but I can definitely see it happening. I think most people know that Git isn't perfect, but it's also the case that coming up with anything better is an extremely difficult task. If it wasn't, someone would have already done it. It's not like people haven't been trying.

AndriyKunitsyn 7y ago

The usual technique of game studios is using Perforce. It is clunky and sometimes straight up infuriating, but it handles large files well.

zeofig 7y ago

Yeah, git with binary files is not fun. Have you tried git annex?

j88439h84 7y ago

What does your layer do?

erikbye 7y ago

Why did you not use Plastic SCM or Perforce?

1auralynn 7y ago
I can't recommend Plastic enough - it's so fast and a great UI but super powerful. I've been using it at my company for years.

DaiPlusPlus 7y ago

Aren’t game studios and other creative studios meant to use “asset management” systems instead for their large binaries? Diffing a PSD as a binary is impossible - whereas proper asset management tools will deconstruct the PSD’s format to make for a human-readable diff (e.g. added/removed layers, properties, etc).

lodi 7y ago
Separate version control for code vs assets leads to a world of pain. Also you can use whatever diff tool you want; doesn't have to be the built-in textual diff.