5 ms·

GitLab Annex: Large Binaries with Git

47 points by jeremiep 12y ago

jobvandervoort 12y ago
GitLab engineer here, let me know if you have any questions. We're quite excited about GitLab Annex and curious to hear what people think about it.
michaelmior 12y ago
Duplicate of https://news.ycombinator.com/item?id=9063705 https://news.ycombinator.com/item?id=9063705
alimoeeny 12y ago
I am trying to understand why putting large binaries under source control is a good idea in the first place. Unless you have a way to make sense of the diffs this will not help right? and to to make sense of the diffs, you need to understand the structure/format of the binary at least after you have the diff, or maybe even to make a useful diff, right?
- barkingcat 12y ago
  Diffing is just one small part of version control. Keeping binary files in version control is the number one step away from messy file-name-based versioning. Many design studios are forced to use project1.ver1.psd project1.ver2.psd project1.ver3.psd, etc and so on in order to version their files. Single psd files can be on the order of high hundreds of megabytes to low gigabytes for high resolution ready-for-press files. Not being able to diff the files is not a problem from an organisational point of view. Of course, in an ideal world there would be diffing of large binaries in a way that makes sense, but thinking there's no use in versioning binaries is very short sighted.
  - jeremiep 12y ago
    This exactly. We keep our game's asset in SVN at the moment only for the convenience of versioning. Artists are still locking files to ensure no conflicts are happening (and therefore no merges are needed). It's much easier to naively do a repository checkout/update than manually detecting changes (or rolling your own solution using rsync or similar). Especially when considering the game has over 100Gb of raw assets. SVN, Git or Perforce might not be the best tools for such a task but it works great.
- jewel 12y ago
  You're right that you can't get a diff of a large binary (or even a small one). It's still useful to know when the file changed and who changed it. For example, imaging you work at a video production company where you have videos on each page of your website. (Let's also imagine that you're not using vimeo or youtube to host them.) You might make a branch of the home page with a different video, and you want to coordinate changes across pages. git-annex is amazing for this use case, as you don't have to actually have a local copy of videos that you're no longer using.
- noselasd 12y ago
  Where would you put them (honest question) ? If you're housing a large part of assets, be it large 3D models, CAD models, music or the like that you're developing, you'd want some place to at least store, version, comment, track them as you're developing and changing that content.
- rmc 12y ago
  I keep my photos in git-annex. It means I have a backup of my photos. If I edit one, the old version is still there. I can "fork" the directories and do some experimental stuff, and switch back to the original version if I need to. Git annex, like git, uses content based hashes for names. The content is preserved. I can copy files to/from remote git annex repos. Easy backup and redundancy.
- deathanatos 12y ago
  I've always belived that source control was for source. Source is any piece of data created by a human being, which includes things such as large binaries, such as psd files (mentioned in a sibling). This belief excludes, however, large binaries output by a build system: the build script is the source (and should get versioned), the output, not so much. (The output should likely get stored somewhere, however. We keep our build scripts in git, and binaries in S3, for example.)
- DougN7 12y ago
  It seems people disagree with me, but checking in the source, the binary DLLs, and debug symbols and tagging them all is fantastic. If we get a crash dump from the field, a quick checkout gives us exactly what we need to diagnose the problem. Everything in one place and perfectly synced. I'm not even sure it's possible for a rebuild to produce exactly the same binary down to the byte (internal versions, timestamps, etc). And even if it could, it would require guaranteeing the build system hadn't been changed, patched, etc. I can only assume people that don't commit their binaries don't have to look at crash dumps very often...?
flyingyeti 12y ago
What is the difference between GitLab Annex and git-annex? https://git-annex.branchable.com/ https://git-annex.branchable.com/ Edit: Just found the relevant quote from the article > In GitLab 7.8 Enterprise Edition this problem is solved by integrating the awesome git-annex. I guess the repeated branding of "GitLab Annex" just seems a bit strange to me.
- deleted 12y ago
  [deleted]
- jobvandervoort 12y ago
  GitLab Annex is git-annex integrated in GitLab. Meaning you have the authentication and protection that GitLab gives you with git-annex.
icebraining 12y ago
Joey Hess mentioned this in the git-annex devlog: http://git-annex.branchable.com/devblog/day_256__sqlite_concurrency_argh/ http://git-annex.branchable.com/devblog/day_256__sqlite_conc...
bitwize 12y ago
Most grownup software shops use Perforce precisely because it can handle large repositories with large amounts of non-source non-textual content. Maybe this will bring Git in the same league with the big boys.
- Ysx 12y ago
  Your condescension must be an unending source of rich and gratifying relationships.
- jobvandervoort 12y ago
  We believe source control should be everywhere and git isn't great yet to work with large binaries. Git(Lab) Annex makes things much better for people working with these kinds of repositories. We'd love to get feedback on it.
  - Jare 12y ago
    As I mentioned in another thread, efficient storage of these files is necessary, but not sufficient. Exclusive locking is a requirement for working with editable (but not mergeable) binary files in a team environment. It's the reason most games studios use Perforce or SVN instead of git. Tracking locks using a separate system does not scale, it needs to be integrated and enforced by the VCS. PlasticSCM has a hybrid approach that is worth studying.
    - sytse 12y ago
      Locking is indeed nice to have. We're considering adding locking to the web interface of GitLab. But we'll probably wait for a customer to sponsor this.
    - deleted 12y ago
      [deleted]
    - lucaspiller 12y ago
      Woukd you mind explaining a bit more why locking is a requirement? It sounds more like an organisational issue rather than something required in SCM. Taking the game studio example, why would two developers/artists/whatever need to work on the same asset at the same time? Locking stops them from stepping on each other's toes, but it also means one has to wait for the other to finish their task before they can do anything.

47 points by jeremiep 12y ago

GitLab engineer here, let me know if you have any questions. We're quite excited about GitLab Annex and curious to hear what people think about it.

michaelmior 12y ago

Duplicate of https://news.ycombinator.com/item?id=9063705 https://news.ycombinator.com/item?id=9063705

alimoeeny 12y ago

I am trying to understand why putting large binaries under source control is a good idea in the first place. Unless you have a way to make sense of the diffs this will not help right? and to to make sense of the diffs, you need to understand the structure/format of the binary at least after you have the diff, or maybe even to make a useful diff, right?

barkingcat 12y ago
Diffing is just one small part of version control. Keeping binary files in version control is the number one step away from messy file-name-based versioning. Many design studios are forced to use project1.ver1.psd project1.ver2.psd project1.ver3.psd, etc and so on in order to version their files. Single psd files can be on the order of high hundreds of megabytes to low gigabytes for high resolution ready-for-press files. Not being able to diff the files is not a problem from an organisational point of view. Of course, in an ideal world there would be diffing of large binaries in a way that makes sense, but thinking there's no use in versioning binaries is very short sighted.
- jeremiep 12y ago
  This exactly. We keep our game's asset in SVN at the moment only for the convenience of versioning. Artists are still locking files to ensure no conflicts are happening (and therefore no merges are needed). It's much easier to naively do a repository checkout/update than manually detecting changes (or rolling your own solution using rsync or similar). Especially when considering the game has over 100Gb of raw assets. SVN, Git or Perforce might not be the best tools for such a task but it works great.
jewel 12y ago
You're right that you can't get a diff of a large binary (or even a small one). It's still useful to know when the file changed and who changed it. For example, imaging you work at a video production company where you have videos on each page of your website. (Let's also imagine that you're not using vimeo or youtube to host them.) You might make a branch of the home page with a different video, and you want to coordinate changes across pages. git-annex is amazing for this use case, as you don't have to actually have a local copy of videos that you're no longer using.
noselasd 12y ago
Where would you put them (honest question) ? If you're housing a large part of assets, be it large 3D models, CAD models, music or the like that you're developing, you'd want some place to at least store, version, comment, track them as you're developing and changing that content.
rmc 12y ago
I keep my photos in git-annex. It means I have a backup of my photos. If I edit one, the old version is still there. I can "fork" the directories and do some experimental stuff, and switch back to the original version if I need to. Git annex, like git, uses content based hashes for names. The content is preserved. I can copy files to/from remote git annex repos. Easy backup and redundancy.
deathanatos 12y ago
I've always belived that source control was for source. Source is any piece of data created by a human being, which includes things such as large binaries, such as psd files (mentioned in a sibling). This belief excludes, however, large binaries output by a build system: the build script is the source (and should get versioned), the output, not so much. (The output should likely get stored somewhere, however. We keep our build scripts in git, and binaries in S3, for example.)
DougN7 12y ago
It seems people disagree with me, but checking in the source, the binary DLLs, and debug symbols and tagging them all is fantastic. If we get a crash dump from the field, a quick checkout gives us exactly what we need to diagnose the problem. Everything in one place and perfectly synced. I'm not even sure it's possible for a rebuild to produce exactly the same binary down to the byte (internal versions, timestamps, etc). And even if it could, it would require guaranteeing the build system hadn't been changed, patched, etc. I can only assume people that don't commit their binaries don't have to look at crash dumps very often...?

flyingyeti 12y ago

What is the difference between GitLab Annex and git-annex? https://git-annex.branchable.com/ https://git-annex.branchable.com/ Edit: Just found the relevant quote from the article > In GitLab 7.8 Enterprise Edition this problem is solved by integrating the awesome git-annex. I guess the repeated branding of "GitLab Annex" just seems a bit strange to me.

deleted 12y ago
[deleted]
jobvandervoort 12y ago
GitLab Annex is git-annex integrated in GitLab. Meaning you have the authentication and protection that GitLab gives you with git-annex.

icebraining 12y ago

Joey Hess mentioned this in the git-annex devlog: http://git-annex.branchable.com/devblog/day_256__sqlite_concurrency_argh/ http://git-annex.branchable.com/devblog/day_256__sqlite_conc...

bitwize 12y ago

Most grownup software shops use Perforce precisely because it can handle large repositories with large amounts of non-source non-textual content. Maybe this will bring Git in the same league with the big boys.

Ysx 12y ago
Your condescension must be an unending source of rich and gratifying relationships.
jobvandervoort 12y ago
We believe source control should be everywhere and git isn't great yet to work with large binaries. Git(Lab) Annex makes things much better for people working with these kinds of repositories. We'd love to get feedback on it.
- Jare 12y ago
  As I mentioned in another thread, efficient storage of these files is necessary, but not sufficient. Exclusive locking is a requirement for working with editable (but not mergeable) binary files in a team environment. It's the reason most games studios use Perforce or SVN instead of git. Tracking locks using a separate system does not scale, it needs to be integrated and enforced by the VCS. PlasticSCM has a hybrid approach that is worth studying.
  - sytse 12y ago
    Locking is indeed nice to have. We're considering adding locking to the web interface of GitLab. But we'll probably wait for a customer to sponsor this.
  - deleted 12y ago
    [deleted]
  - lucaspiller 12y ago
    Woukd you mind explaining a bit more why locking is a requirement? It sounds more like an organisational issue rather than something required in SCM. Taking the game studio example, why would two developers/artists/whatever need to work on the same asset at the same time? Locking stops them from stepping on each other's toes, but it also means one has to wait for the other to finish their task before they can do anything.