4 ms·

How is it possible that nobody has dealt with this problem yet?

by naner 17y ago

pmjordan 17y ago
Active open source development rarely meets "AAA" game development.
- sshumaker 17y ago
  One could argue that the traditional source control model (where everyone has a copy on their own machine) isn't a great way to deal with huge binary assets. I've worked on game projects where the binary churn was on the order of gigabytes a day. Even though everyone was physically co-located, when everyone arrived at 10am and started to update to head, a hundred computers all trying to download the days worth of 2 gigs of assets over the network didn't make for a pretty picture. You can try to cronjob the updates, staggered around 4am. Or you can realize that the majority of people are only modifying a subset of these assets, and roll your own link-based asset manager for large binaries.
  - pmjordan 17y ago
    Yes, absolutely, you wouldn't want everyone to keep pulling all data, git-style. The programmers usually only need the assets in the game-ready format, not the source models, textures, etc. which are typically much larger, and only needed by the artists/level designers working with them.
ableal 17y ago
I have been trying to point it out [1], without much luck. Today I got in early by chance, and decided on the Socratic approach ;-) [1] http://news.ycombinator.com/item?id=1201559 http://news.ycombinator.com/item?id=1201559, http://news.ycombinator.com/item?id=1219082 http://news.ycombinator.com/item?id=1219082, http://news.ycombinator.com/item?id=1222905 http://news.ycombinator.com/item?id=1222905, http://news.ycombinator.com/item?id=1242374 http://news.ycombinator.com/item?id=1242374
blasdel 17y ago
Because anyone cleverer than a doorknob uses the right tool for the job: rsync. They're almost always artifacts, not assets. You handle versioning either through filesystem snapshots, or by just using different paths for each version.
- ableal 17y ago
  So, git/hg/etc are doorknobs ? Of course not, it's not their problem. What annoys me is that the "You must use DVCS" memo is being sent around uncritically, and anyone who does not fit the use case is left out. I think that people who deal with images/spreadsheets/data would also benefit from a good VCS tool. If the coders, having scratched their itch, did not declare the problem solved and move on ... P.S. thanks for the polite answer. So, there is either no problem or no solution ... I think we're done here.
  - blasdel 17y ago
    git/hg/etc are not doorknobs. My simile was comparing you, and people like you, to doorknobs. http://www.google.com/search?q=dumber+than+a+doorknob http://www.google.com/search?q=dumber+than+a+doorknob No version control system, centralized or distributed, actually handles blobs better than just treating them as artifacts and using rsync. The only way to actually solve the problem is to make the data not be blobs anymore, where the diff/merge/serialization code all understands at least the structural container format and can render it usefully. There's never going to be a general purpose tool that does that outside of a live-in smalltalk image (or similar). The best you're going to get is tools vertically-integrated with the application, and all the ones I've seen to date (MS Office, Adobe Version Cue, etc.) all do a terrible job of even doing diffs/RCS, much less actually implementing a real VCS.
stevelosh 17y ago
Greg Ward and some others are working on it with the bfiles extension for Mercurial. It's a work in progress but at least it's getting the ball rolling: http://mercurial.selenic.com/wiki/BfilesExtension http://mercurial.selenic.com/wiki/BfilesExtension
- ableal 17y ago
  Thank you very much. I liked what I read in the design doc (http://vc.gerg.ca/hg/hg-bfiles/raw-file/tip/design.txt http://vc.gerg.ca/hg/hg-bfiles/raw-file/tip/design.txt), which seems to be trying to automatically do the best thing possible under the hg constraints. It's interesting that the names are "Distributed Version Control Systems", or "Revision Control", but the ghost of the word "source" (meaning code text, in 1972's SCCS) is still read into it. Surprisingly, when DVCS are shouted as the bees' knees for everybody, people who have images, documents and data may think they were included. Apparently not, those benighted heathens should use rsync or whatever manually, and not soil the "source control" systems with their binaries. Or go find not-included plug-ins, if they can. (Those people are apparently too dumb to understand that a binary "artifact" produced with an image editor, spreadsheet, or data analysis tool is fundamentally different from a blessed code file written with a text editor, and thus is not entitled to be under real version control, or have a DCVS tool automatically do something smart about it.) The Perforce guys must be laughing their asses off.

How is it possible that nobody has dealt with this problem yet?

by naner 17y ago

How is it possible that nobody has dealt with this problem yet?

pmjordan 17y ago

Active open source development rarely meets "AAA" game development.

sshumaker 17y ago
One could argue that the traditional source control model (where everyone has a copy on their own machine) isn't a great way to deal with huge binary assets. I've worked on game projects where the binary churn was on the order of gigabytes a day. Even though everyone was physically co-located, when everyone arrived at 10am and started to update to head, a hundred computers all trying to download the days worth of 2 gigs of assets over the network didn't make for a pretty picture. You can try to cronjob the updates, staggered around 4am. Or you can realize that the majority of people are only modifying a subset of these assets, and roll your own link-based asset manager for large binaries.
- pmjordan 17y ago
  Yes, absolutely, you wouldn't want everyone to keep pulling all data, git-style. The programmers usually only need the assets in the game-ready format, not the source models, textures, etc. which are typically much larger, and only needed by the artists/level designers working with them.

ableal 17y ago

I have been trying to point it out [1], without much luck. Today I got in early by chance, and decided on the Socratic approach ;-) [1] http://news.ycombinator.com/item?id=1201559 http://news.ycombinator.com/item?id=1201559, http://news.ycombinator.com/item?id=1219082 http://news.ycombinator.com/item?id=1219082, http://news.ycombinator.com/item?id=1222905 http://news.ycombinator.com/item?id=1222905, http://news.ycombinator.com/item?id=1242374 http://news.ycombinator.com/item?id=1242374

blasdel 17y ago

Because anyone cleverer than a doorknob uses the right tool for the job: rsync. They're almost always artifacts, not assets. You handle versioning either through filesystem snapshots, or by just using different paths for each version.

ableal 17y ago
So, git/hg/etc are doorknobs ? Of course not, it's not their problem. What annoys me is that the "You must use DVCS" memo is being sent around uncritically, and anyone who does not fit the use case is left out. I think that people who deal with images/spreadsheets/data would also benefit from a good VCS tool. If the coders, having scratched their itch, did not declare the problem solved and move on ... P.S. thanks for the polite answer. So, there is either no problem or no solution ... I think we're done here.
- blasdel 17y ago
  git/hg/etc are not doorknobs. My simile was comparing you, and people like you, to doorknobs. http://www.google.com/search?q=dumber+than+a+doorknob http://www.google.com/search?q=dumber+than+a+doorknob No version control system, centralized or distributed, actually handles blobs better than just treating them as artifacts and using rsync. The only way to actually solve the problem is to make the data not be blobs anymore, where the diff/merge/serialization code all understands at least the structural container format and can render it usefully. There's never going to be a general purpose tool that does that outside of a live-in smalltalk image (or similar). The best you're going to get is tools vertically-integrated with the application, and all the ones I've seen to date (MS Office, Adobe Version Cue, etc.) all do a terrible job of even doing diffs/RCS, much less actually implementing a real VCS.

stevelosh 17y ago

Greg Ward and some others are working on it with the bfiles extension for Mercurial. It's a work in progress but at least it's getting the ball rolling: http://mercurial.selenic.com/wiki/BfilesExtension http://mercurial.selenic.com/wiki/BfilesExtension

ableal 17y ago
Thank you very much. I liked what I read in the design doc (http://vc.gerg.ca/hg/hg-bfiles/raw-file/tip/design.txt http://vc.gerg.ca/hg/hg-bfiles/raw-file/tip/design.txt), which seems to be trying to automatically do the best thing possible under the hg constraints. It's interesting that the names are "Distributed Version Control Systems", or "Revision Control", but the ghost of the word "source" (meaning code text, in 1972's SCCS) is still read into it. Surprisingly, when DVCS are shouted as the bees' knees for everybody, people who have images, documents and data may think they were included. Apparently not, those benighted heathens should use rsync or whatever manually, and not soil the "source control" systems with their binaries. Or go find not-included plug-ins, if they can. (Those people are apparently too dumb to understand that a binary "artifact" produced with an image editor, spreadsheet, or data analysis tool is fundamentally different from a blessed code file written with a text editor, and thus is not entitled to be under real version control, or have a DCVS tool automatically do something smart about it.) The Perforce guys must be laughing their asses off.