4 ms·

SVN works better when you're dealing with large binary files. If you're bundling in all of your dependencies, or some images/models/sound files then Git gets v

SVN works better when you're dealing with large binary files. If you're bundling in all of your dependencies, or some images/models/sound files then Git gets very big, very quickly.

dsirijus 13y ago
Also, a memory hungry monster, while it's forwarding them.
jblow 13y ago
Yes, and I am surprised that this didn't up as answer #1 in response to the original question. This problem is an absolute showstopper if you need to manage binary assets (which for something like video games, you absolutely do -- you want to know that your data is the right data for the current version of the code, because the data is changing all the time too). In The Witness (http://the-witness.net/news http://the-witness.net/news) we have 20GB of data checked into svn. Try that with git.
- AbsoluteDestiny 13y ago
  You have to use something like git-annex. Possibly some other options that others might know of.
  - fsckin 13y ago
    Using git-annex would probably fix a lot of these problems. On the other hand, Perforce is the game industry standard. As much as engineers complain how terrible P4 is in comparison to distributed options, the other teams working on the title (artists, designers, QA/Test) save a huge amount of time by simply not fucking things up for everyone else. When there's 250 people working on a title, producers will call to lock the entire depot down so that Intern Joe Blow from Test/QA can't check in a broken build and waste the time of everyone who is working on the project. I've administered depots (along with overseas proxies) with a 1TB head revision. It was amazingly fast despite how huge the data set was. The only exception was syncing a new workspace, which is easy enough to work around with weekly snapshots and rsync.
- __--__ 13y ago
  Try that with git. My last company did just that. The repository wound up being 30gb at the end of the project (which was a 3d facebook game). We tried to host on a company hosted github, but got kicked off because it couldn't deal with a repository that size[1]. Luckily, we only had one person try to check out the repository at the end of the project. It took them 14 hours for the initial check out. [1] It was never made clear to me whether it was our hardware that had the problem or the github self hosted software. Either way, it was unreasonably slow.
- rmc 13y ago
  Use git-annex. I have 100gb of video footage using git annex.
andrewflnr 13y ago
How does mercurial do in that situation? I know it uses a binary diff algorithm under the hood, but are there other showstoppers for storing giant files in mercurial?
- aardvark179 13y ago
  They don't store well as compressed binary diffs, and they can cause high memory consumption during various operations. If you absolutely must store large binary files then there's a large files extension.
  - e12e 13y ago
    The largefiles extention is now bundled with mercurial (since 2.0): http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LargefilesExtension [edit: this isn't quite a solution, more of a work around -- but one that can be expected to be installed by default.]
patrickg 13y ago
`git gc` packs files with delta compression. I have a rather big git repository (daily mirroring a 25 gig repository with some large dvd iso images). And it seems to work quite well.
lightblade 13y ago
This just happened at work like 2 weeks ago. Someone checked in a large (4g) binary file into SVN repo and it crashed the whole system. Everyone at the company was blocked for half a day before they got it fixed (temporarily to unblock everyone). But it took another 2 days and a lot of heated debates between the release manager and devops before everything went back to normal. There were talks about moving to git before the incident, but this really pushed management to finally pull the trigger. 2 lessons here: 1. SVN is bad at large binary files; 2) centralized source control's one way of doing things also gives you one way of disaster recovery.
- AndrewDucker 13y ago
  I'm not surprised it choked on a 4Gig binary file. But when we have dozens of multi-MB binary files, which change every week or so, then after a year we're ending up with a massive repository to pull down.

SVN works better when you're dealing with large binary files. If you're bundling in all of your dependencies, or some images/models/sound files then Git gets very big, very quickly.

dsirijus 13y ago

Also, a memory hungry monster, while it's forwarding them.

jblow 13y ago

Yes, and I am surprised that this didn't up as answer #1 in response to the original question. This problem is an absolute showstopper if you need to manage binary assets (which for something like video games, you absolutely do -- you want to know that your data is the right data for the current version of the code, because the data is changing all the time too). In The Witness (http://the-witness.net/news http://the-witness.net/news) we have 20GB of data checked into svn. Try that with git.

AbsoluteDestiny 13y ago
You have to use something like git-annex. Possibly some other options that others might know of.
- fsckin 13y ago
  Using git-annex would probably fix a lot of these problems. On the other hand, Perforce is the game industry standard. As much as engineers complain how terrible P4 is in comparison to distributed options, the other teams working on the title (artists, designers, QA/Test) save a huge amount of time by simply not fucking things up for everyone else. When there's 250 people working on a title, producers will call to lock the entire depot down so that Intern Joe Blow from Test/QA can't check in a broken build and waste the time of everyone who is working on the project. I've administered depots (along with overseas proxies) with a 1TB head revision. It was amazingly fast despite how huge the data set was. The only exception was syncing a new workspace, which is easy enough to work around with weekly snapshots and rsync.
__--__ 13y ago
Try that with git. My last company did just that. The repository wound up being 30gb at the end of the project (which was a 3d facebook game). We tried to host on a company hosted github, but got kicked off because it couldn't deal with a repository that size[1]. Luckily, we only had one person try to check out the repository at the end of the project. It took them 14 hours for the initial check out. [1] It was never made clear to me whether it was our hardware that had the problem or the github self hosted software. Either way, it was unreasonably slow.
rmc 13y ago
Use git-annex. I have 100gb of video footage using git annex.

andrewflnr 13y ago

How does mercurial do in that situation? I know it uses a binary diff algorithm under the hood, but are there other showstoppers for storing giant files in mercurial?

aardvark179 13y ago
They don't store well as compressed binary diffs, and they can cause high memory consumption during various operations. If you absolutely must store large binary files then there's a large files extension.
- e12e 13y ago
  The largefiles extention is now bundled with mercurial (since 2.0): http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LargefilesExtension [edit: this isn't quite a solution, more of a work around -- but one that can be expected to be installed by default.]

patrickg 13y ago

`git gc` packs files with delta compression. I have a rather big git repository (daily mirroring a 25 gig repository with some large dvd iso images). And it seems to work quite well.

lightblade 13y ago

This just happened at work like 2 weeks ago. Someone checked in a large (4g) binary file into SVN repo and it crashed the whole system. Everyone at the company was blocked for half a day before they got it fixed (temporarily to unblock everyone). But it took another 2 days and a lot of heated debates between the release manager and devops before everything went back to normal. There were talks about moving to git before the incident, but this really pushed management to finally pull the trigger. 2 lessons here: 1. SVN is bad at large binary files; 2) centralized source control's one way of doing things also gives you one way of disaster recovery.

AndrewDucker 13y ago
I'm not surprised it choked on a 4Gig binary file. But when we have dozens of multi-MB binary files, which change every week or so, then after a year we're ending up with a massive repository to pull down.