4 ms·

I am trying to understand why putting large binaries under source control is a good idea in the first place. Unless you have a way to make sense of the diffs

I am trying to understand why putting large binaries under source control is a good idea in the first place. Unless you have a way to make sense of the diffs this will not help right? and to to make sense of the diffs, you need to understand the structure/format of the binary at least after you have the diff, or maybe even to make a useful diff, right?

barkingcat 12y ago
Diffing is just one small part of version control. Keeping binary files in version control is the number one step away from messy file-name-based versioning. Many design studios are forced to use project1.ver1.psd project1.ver2.psd project1.ver3.psd, etc and so on in order to version their files. Single psd files can be on the order of high hundreds of megabytes to low gigabytes for high resolution ready-for-press files. Not being able to diff the files is not a problem from an organisational point of view. Of course, in an ideal world there would be diffing of large binaries in a way that makes sense, but thinking there's no use in versioning binaries is very short sighted.
- jeremiep 12y ago
  This exactly. We keep our game's asset in SVN at the moment only for the convenience of versioning. Artists are still locking files to ensure no conflicts are happening (and therefore no merges are needed). It's much easier to naively do a repository checkout/update than manually detecting changes (or rolling your own solution using rsync or similar). Especially when considering the game has over 100Gb of raw assets. SVN, Git or Perforce might not be the best tools for such a task but it works great.
jewel 12y ago
You're right that you can't get a diff of a large binary (or even a small one). It's still useful to know when the file changed and who changed it. For example, imaging you work at a video production company where you have videos on each page of your website. (Let's also imagine that you're not using vimeo or youtube to host them.) You might make a branch of the home page with a different video, and you want to coordinate changes across pages. git-annex is amazing for this use case, as you don't have to actually have a local copy of videos that you're no longer using.
noselasd 12y ago
Where would you put them (honest question) ? If you're housing a large part of assets, be it large 3D models, CAD models, music or the like that you're developing, you'd want some place to at least store, version, comment, track them as you're developing and changing that content.
rmc 12y ago
I keep my photos in git-annex. It means I have a backup of my photos. If I edit one, the old version is still there. I can "fork" the directories and do some experimental stuff, and switch back to the original version if I need to. Git annex, like git, uses content based hashes for names. The content is preserved. I can copy files to/from remote git annex repos. Easy backup and redundancy.
deathanatos 12y ago
I've always belived that source control was for source. Source is any piece of data created by a human being, which includes things such as large binaries, such as psd files (mentioned in a sibling). This belief excludes, however, large binaries output by a build system: the build script is the source (and should get versioned), the output, not so much. (The output should likely get stored somewhere, however. We keep our build scripts in git, and binaries in S3, for example.)
DougN7 12y ago
It seems people disagree with me, but checking in the source, the binary DLLs, and debug symbols and tagging them all is fantastic. If we get a crash dump from the field, a quick checkout gives us exactly what we need to diagnose the problem. Everything in one place and perfectly synced. I'm not even sure it's possible for a rebuild to produce exactly the same binary down to the byte (internal versions, timestamps, etc). And even if it could, it would require guaranteeing the build system hadn't been changed, patched, etc. I can only assume people that don't commit their binaries don't have to look at crash dumps very often...?

barkingcat 12y ago

Diffing is just one small part of version control. Keeping binary files in version control is the number one step away from messy file-name-based versioning. Many design studios are forced to use project1.ver1.psd project1.ver2.psd project1.ver3.psd, etc and so on in order to version their files. Single psd files can be on the order of high hundreds of megabytes to low gigabytes for high resolution ready-for-press files. Not being able to diff the files is not a problem from an organisational point of view. Of course, in an ideal world there would be diffing of large binaries in a way that makes sense, but thinking there's no use in versioning binaries is very short sighted.

jeremiep 12y ago
This exactly. We keep our game's asset in SVN at the moment only for the convenience of versioning. Artists are still locking files to ensure no conflicts are happening (and therefore no merges are needed). It's much easier to naively do a repository checkout/update than manually detecting changes (or rolling your own solution using rsync or similar). Especially when considering the game has over 100Gb of raw assets. SVN, Git or Perforce might not be the best tools for such a task but it works great.

jewel 12y ago

You're right that you can't get a diff of a large binary (or even a small one). It's still useful to know when the file changed and who changed it. For example, imaging you work at a video production company where you have videos on each page of your website. (Let's also imagine that you're not using vimeo or youtube to host them.) You might make a branch of the home page with a different video, and you want to coordinate changes across pages. git-annex is amazing for this use case, as you don't have to actually have a local copy of videos that you're no longer using.

noselasd 12y ago

Where would you put them (honest question) ? If you're housing a large part of assets, be it large 3D models, CAD models, music or the like that you're developing, you'd want some place to at least store, version, comment, track them as you're developing and changing that content.

rmc 12y ago

I keep my photos in git-annex. It means I have a backup of my photos. If I edit one, the old version is still there. I can "fork" the directories and do some experimental stuff, and switch back to the original version if I need to. Git annex, like git, uses content based hashes for names. The content is preserved. I can copy files to/from remote git annex repos. Easy backup and redundancy.

deathanatos 12y ago

I've always belived that source control was for source. Source is any piece of data created by a human being, which includes things such as large binaries, such as psd files (mentioned in a sibling). This belief excludes, however, large binaries output by a build system: the build script is the source (and should get versioned), the output, not so much. (The output should likely get stored somewhere, however. We keep our build scripts in git, and binaries in S3, for example.)

DougN7 12y ago

It seems people disagree with me, but checking in the source, the binary DLLs, and debug symbols and tagging them all is fantastic. If we get a crash dump from the field, a quick checkout gives us exactly what we need to diagnose the problem. Everything in one place and perfectly synced. I'm not even sure it's possible for a rebuild to produce exactly the same binary down to the byte (internal versions, timestamps, etc). And even if it could, it would require guaranteeing the build system hadn't been changed, patched, etc. I can only assume people that don't commit their binaries don't have to look at crash dumps very often...?