4 ms·

Poor or inefficient support for large binary assets, such as compiled objects, images, or video. Especially in the context of video game development, binary ass

Poor or inefficient support for large binary assets, such as compiled objects, images, or video. Especially in the context of video game development, binary assets that constantly change are highly painful to deal with in Git or Mercurial.

naner 17y ago
How is it possible that nobody has dealt with this problem yet?
- pmjordan 17y ago
  Active open source development rarely meets "AAA" game development.
  - sshumaker 17y ago
    One could argue that the traditional source control model (where everyone has a copy on their own machine) isn't a great way to deal with huge binary assets. I've worked on game projects where the binary churn was on the order of gigabytes a day. Even though everyone was physically co-located, when everyone arrived at 10am and started to update to head, a hundred computers all trying to download the days worth of 2 gigs of assets over the network didn't make for a pretty picture. You can try to cronjob the updates, staggered around 4am. Or you can realize that the majority of people are only modifying a subset of these assets, and roll your own link-based asset manager for large binaries.
    - pmjordan 17y ago
      Yes, absolutely, you wouldn't want everyone to keep pulling all data, git-style. The programmers usually only need the assets in the game-ready format, not the source models, textures, etc. which are typically much larger, and only needed by the artists/level designers working with them.
- ableal 17y ago
  I have been trying to point it out [1], without much luck. Today I got in early by chance, and decided on the Socratic approach ;-) [1] http://news.ycombinator.com/item?id=1201559 http://news.ycombinator.com/item?id=1201559, http://news.ycombinator.com/item?id=1219082 http://news.ycombinator.com/item?id=1219082, http://news.ycombinator.com/item?id=1222905 http://news.ycombinator.com/item?id=1222905, http://news.ycombinator.com/item?id=1242374 http://news.ycombinator.com/item?id=1242374
- blasdel 17y ago
  Because anyone cleverer than a doorknob uses the right tool for the job: rsync. They're almost always artifacts, not assets. You handle versioning either through filesystem snapshots, or by just using different paths for each version.
  - ableal 17y ago
    So, git/hg/etc are doorknobs ? Of course not, it's not their problem. What annoys me is that the "You must use DVCS" memo is being sent around uncritically, and anyone who does not fit the use case is left out. I think that people who deal with images/spreadsheets/data would also benefit from a good VCS tool. If the coders, having scratched their itch, did not declare the problem solved and move on ... P.S. thanks for the polite answer. So, there is either no problem or no solution ... I think we're done here.
    - blasdel 17y ago
      git/hg/etc are not doorknobs. My simile was comparing you, and people like you, to doorknobs. http://www.google.com/search?q=dumber+than+a+doorknob http://www.google.com/search?q=dumber+than+a+doorknob No version control system, centralized or distributed, actually handles blobs better than just treating them as artifacts and using rsync. The only way to actually solve the problem is to make the data not be blobs anymore, where the diff/merge/serialization code all understands at least the structural container format and can render it usefully. There's never going to be a general purpose tool that does that outside of a live-in smalltalk image (or similar). The best you're going to get is tools vertically-integrated with the application, and all the ones I've seen to date (MS Office, Adobe Version Cue, etc.) all do a terrible job of even doing diffs/RCS, much less actually implementing a real VCS.
- stevelosh 17y ago
  Greg Ward and some others are working on it with the bfiles extension for Mercurial. It's a work in progress but at least it's getting the ball rolling: http://mercurial.selenic.com/wiki/BfilesExtension http://mercurial.selenic.com/wiki/BfilesExtension
  - ableal 17y ago
    Thank you very much. I liked what I read in the design doc (http://vc.gerg.ca/hg/hg-bfiles/raw-file/tip/design.txt http://vc.gerg.ca/hg/hg-bfiles/raw-file/tip/design.txt), which seems to be trying to automatically do the best thing possible under the hg constraints. It's interesting that the names are "Distributed Version Control Systems", or "Revision Control", but the ghost of the word "source" (meaning code text, in 1972's SCCS) is still read into it. Surprisingly, when DVCS are shouted as the bees' knees for everybody, people who have images, documents and data may think they were included. Apparently not, those benighted heathens should use rsync or whatever manually, and not soil the "source control" systems with their binaries. Or go find not-included plug-ins, if they can. (Those people are apparently too dumb to understand that a binary "artifact" produced with an image editor, spreadsheet, or data analysis tool is fundamentally different from a blessed code file written with a text editor, and thus is not entitled to be under real version control, or have a DCVS tool automatically do something smart about it.) The Perforce guys must be laughing their asses off.
lnguyen 17y ago
It's not limited to DVCS or open source. Commercial tools such as ClearCase, AllFusion/Harvest, and Dimensions still handle large binary assets poorly. I don't think you'll find a satisfactory tool that has as its primary use the management of texts and deltas of texts. You'd probably need to use a combination of tools which has its own set of frustrations.
- smallblacksun 17y ago
  Perforce works well with large binaries and text (costs a fortune, though).
- sandGorgon 17y ago
  Bitkeeper (the SCM that git was "inspired" from), claims good support for large binary assets BAM solves this problem with a hybrid approach. BAM adds the concept of one or more BAM servers. Each BAM server contain all BAM data, but clones of the server (work spaces) contain only that data that the user requests. Typically this is the most recent version of each binary but in some cases it may even be less than that. BAM has been successfully deployed in the game development space with great success. Game developers continue to enjoy the benefits of distributed work flow without the penalty of carrying all versions in every workspace.
sshumaker 17y ago
With binary files, you really want an explicit checkout (exclusive lock) system - since there's typically no way to merge them. This makes the commit/merge style SCSes a bad fit for binary assets (and partially explains the popularity of Perforce, which is a lock-based SCS).
- dalore 17y ago
  That and Perforce is a commercial company that provides support with their product. By default perforce doesn't lock, but it knows when a user has a file checked out for "edit". By default all files are set read only. This was a big downside when you had no internet connection.
  - koenigdavidmj 17y ago
    Yeah, but 'chmod u+w filename' and you're back in business. The GUI has a "resolve offline work" feature that will find everything. Of course, if you don't mark it writable (and just use :w! in Vim, for instance), then a 'p4 sync' may eat your work if anyone's changed the file.
    - dalore 16y ago
      Ahh the resolve offline work feature must be new. It wasn't around when I was using perforce.
      - koenigdavidmj 16y ago
        Sadly, it has no direct CLI equivalent. You may find http://p4delta.sourceforge.net/ http://p4delta.sourceforge.net/ useful, however; it's like `svn up` for Perforce.
prog 17y ago
This is my pet peeve with all DVCSes. I tried this with bzr, hg and git and all fail. Not sure if any other DVCSes handle it. I don't see it happening anytime soon as long as it justified using "its _source_ control". There are programming activities that require large binary files to be versioned along with source and this is a clearly a limitation of DVCSes IMO. Apart from that I am quite happy moving to bzr from svn. Life is much easier now.
ajb 17y ago
Apparently vesta (http://www.vestasys.org http://www.vestasys.org) can handle large binaries (it's from the EDA world, so this is plausible). It's open source, but the extent to which it's distributed is debatable - you need a server, but anyone can install a server and they form a distributed network like a dvcs.
- ableal 17y ago
  Thanks, looks interesting, and it came from DEC via Compaq. It's billed as a Configuration Management System (i.e. managing builds). They do talk of managing tens of GB of derived data, which is also important. The hardware part seems limited: "the Alpha group ... hardware description language files into Vesta's source code control", so not schematics or layout binaries as 'source'. That site seems to be resting since 2006, but there is some activity at http://sourceforge.net/projects/vesta/ http://sourceforge.net/projects/vesta/ (releases in 2009, mailing lists with 2010 messages).
  - ajb 17y ago
    Last time I looked, the IRC channel was somewhat more active than the list.

naner 17y ago

How is it possible that nobody has dealt with this problem yet?

pmjordan 17y ago
Active open source development rarely meets "AAA" game development.
- sshumaker 17y ago
  One could argue that the traditional source control model (where everyone has a copy on their own machine) isn't a great way to deal with huge binary assets. I've worked on game projects where the binary churn was on the order of gigabytes a day. Even though everyone was physically co-located, when everyone arrived at 10am and started to update to head, a hundred computers all trying to download the days worth of 2 gigs of assets over the network didn't make for a pretty picture. You can try to cronjob the updates, staggered around 4am. Or you can realize that the majority of people are only modifying a subset of these assets, and roll your own link-based asset manager for large binaries.
  - pmjordan 17y ago
    Yes, absolutely, you wouldn't want everyone to keep pulling all data, git-style. The programmers usually only need the assets in the game-ready format, not the source models, textures, etc. which are typically much larger, and only needed by the artists/level designers working with them.
ableal 17y ago
I have been trying to point it out [1], without much luck. Today I got in early by chance, and decided on the Socratic approach ;-) [1] http://news.ycombinator.com/item?id=1201559 http://news.ycombinator.com/item?id=1201559, http://news.ycombinator.com/item?id=1219082 http://news.ycombinator.com/item?id=1219082, http://news.ycombinator.com/item?id=1222905 http://news.ycombinator.com/item?id=1222905, http://news.ycombinator.com/item?id=1242374 http://news.ycombinator.com/item?id=1242374
blasdel 17y ago
Because anyone cleverer than a doorknob uses the right tool for the job: rsync. They're almost always artifacts, not assets. You handle versioning either through filesystem snapshots, or by just using different paths for each version.
- ableal 17y ago
  So, git/hg/etc are doorknobs ? Of course not, it's not their problem. What annoys me is that the "You must use DVCS" memo is being sent around uncritically, and anyone who does not fit the use case is left out. I think that people who deal with images/spreadsheets/data would also benefit from a good VCS tool. If the coders, having scratched their itch, did not declare the problem solved and move on ... P.S. thanks for the polite answer. So, there is either no problem or no solution ... I think we're done here.
  - blasdel 17y ago
    git/hg/etc are not doorknobs. My simile was comparing you, and people like you, to doorknobs. http://www.google.com/search?q=dumber+than+a+doorknob http://www.google.com/search?q=dumber+than+a+doorknob No version control system, centralized or distributed, actually handles blobs better than just treating them as artifacts and using rsync. The only way to actually solve the problem is to make the data not be blobs anymore, where the diff/merge/serialization code all understands at least the structural container format and can render it usefully. There's never going to be a general purpose tool that does that outside of a live-in smalltalk image (or similar). The best you're going to get is tools vertically-integrated with the application, and all the ones I've seen to date (MS Office, Adobe Version Cue, etc.) all do a terrible job of even doing diffs/RCS, much less actually implementing a real VCS.
stevelosh 17y ago
Greg Ward and some others are working on it with the bfiles extension for Mercurial. It's a work in progress but at least it's getting the ball rolling: http://mercurial.selenic.com/wiki/BfilesExtension http://mercurial.selenic.com/wiki/BfilesExtension
- ableal 17y ago
  Thank you very much. I liked what I read in the design doc (http://vc.gerg.ca/hg/hg-bfiles/raw-file/tip/design.txt http://vc.gerg.ca/hg/hg-bfiles/raw-file/tip/design.txt), which seems to be trying to automatically do the best thing possible under the hg constraints. It's interesting that the names are "Distributed Version Control Systems", or "Revision Control", but the ghost of the word "source" (meaning code text, in 1972's SCCS) is still read into it. Surprisingly, when DVCS are shouted as the bees' knees for everybody, people who have images, documents and data may think they were included. Apparently not, those benighted heathens should use rsync or whatever manually, and not soil the "source control" systems with their binaries. Or go find not-included plug-ins, if they can. (Those people are apparently too dumb to understand that a binary "artifact" produced with an image editor, spreadsheet, or data analysis tool is fundamentally different from a blessed code file written with a text editor, and thus is not entitled to be under real version control, or have a DCVS tool automatically do something smart about it.) The Perforce guys must be laughing their asses off.

lnguyen 17y ago

It's not limited to DVCS or open source. Commercial tools such as ClearCase, AllFusion/Harvest, and Dimensions still handle large binary assets poorly. I don't think you'll find a satisfactory tool that has as its primary use the management of texts and deltas of texts. You'd probably need to use a combination of tools which has its own set of frustrations.

smallblacksun 17y ago
Perforce works well with large binaries and text (costs a fortune, though).
sandGorgon 17y ago
Bitkeeper (the SCM that git was "inspired" from), claims good support for large binary assets BAM solves this problem with a hybrid approach. BAM adds the concept of one or more BAM servers. Each BAM server contain all BAM data, but clones of the server (work spaces) contain only that data that the user requests. Typically this is the most recent version of each binary but in some cases it may even be less than that. BAM has been successfully deployed in the game development space with great success. Game developers continue to enjoy the benefits of distributed work flow without the penalty of carrying all versions in every workspace.

sshumaker 17y ago

With binary files, you really want an explicit checkout (exclusive lock) system - since there's typically no way to merge them. This makes the commit/merge style SCSes a bad fit for binary assets (and partially explains the popularity of Perforce, which is a lock-based SCS).

dalore 17y ago
That and Perforce is a commercial company that provides support with their product. By default perforce doesn't lock, but it knows when a user has a file checked out for "edit". By default all files are set read only. This was a big downside when you had no internet connection.
- koenigdavidmj 17y ago
  Yeah, but 'chmod u+w filename' and you're back in business. The GUI has a "resolve offline work" feature that will find everything. Of course, if you don't mark it writable (and just use :w! in Vim, for instance), then a 'p4 sync' may eat your work if anyone's changed the file.
  - dalore 16y ago
    Ahh the resolve offline work feature must be new. It wasn't around when I was using perforce.
    - koenigdavidmj 16y ago
      Sadly, it has no direct CLI equivalent. You may find http://p4delta.sourceforge.net/ http://p4delta.sourceforge.net/ useful, however; it's like `svn up` for Perforce.

prog 17y ago

This is my pet peeve with all DVCSes. I tried this with bzr, hg and git and all fail. Not sure if any other DVCSes handle it. I don't see it happening anytime soon as long as it justified using "its _source_ control". There are programming activities that require large binary files to be versioned along with source and this is a clearly a limitation of DVCSes IMO. Apart from that I am quite happy moving to bzr from svn. Life is much easier now.

ajb 17y ago

Apparently vesta (http://www.vestasys.org http://www.vestasys.org) can handle large binaries (it's from the EDA world, so this is plausible). It's open source, but the extent to which it's distributed is debatable - you need a server, but anyone can install a server and they form a distributed network like a dvcs.

ableal 17y ago
Thanks, looks interesting, and it came from DEC via Compaq. It's billed as a Configuration Management System (i.e. managing builds). They do talk of managing tens of GB of derived data, which is also important. The hardware part seems limited: "the Alpha group ... hardware description language files into Vesta's source code control", so not schematics or layout binaries as 'source'. That site seems to be resting since 2006, but there is some activity at http://sourceforge.net/projects/vesta/ http://sourceforge.net/projects/vesta/ (releases in 2009, mailing lists with 2010 messages).
- ajb 17y ago
  Last time I looked, the IRC channel was somewhat more active than the list.