6 ms·

This article doesn't really state the problem precisely; that problem is large data sets. The Linux kernel has a lot of code but weighs in at ~300MB - not even

This article doesn't really state the problem precisely; that problem is large data sets. The Linux kernel has a lot of code but weighs in at ~300MB - not even a gigabyte uncompressed. But if you start checking in lots of binaries - or if you use source control for assets as happens in a game production environment - you start having serious, serious scalability problems because hashing those huge files is no longer fast, and traversing deep directory structures with lots of files becomes a scary problem.. I speak from experience; the game I'm working on at my studio has an in-house tool to optimally paletteize 2d assets for an embedded system. It's a computationally hard problem so a lot of intermediate data is generated to make the final build relatively fast. This tool generates about five files, IIRC, and one folder, for each frame of animation. Multiply this by 30 frames per second for each sprite's anim. Multiply this by an entire game's assets. The result is 80,000+ files and folders checked into SVN. That's after the coder optimized the number of files needed slightly. It can take 20 minutes or more to do an SVN update because of all the checks needed. The solution we really need is a versioning system for binary files. There aren't many of those. Maybe Perforce works. I don't know, I've never used it.

old-gregg 18y ago
We have a very similar problem [except we have videos that need to sit right next to the code tree] and we have added those video subfolders into .gitignore and use rsync for those instead - simple and effective tool-per-task. I don't believe in 6GB source trees without binary blobs. Perhaps Windows codebase is that large but I would assume it's hosted in multiple repos and very rarely is built/checked out by individual developers all at once.
- silentbicycle 18y ago
  > I don't believe in 6GB source trees without binary blobs. No kidding! Even the worst copy-and-paste programming would compress well. (Also, .gitignore + rsync seems like the best option to me, too.)
Semiapies 18y ago
Media assets are definitely an interesting problem. (Of course, then you're beyond "source control" and into something else, but that something else is important.)
windsurfer 18y ago
Git has a feature called "superprojects". You would configure this project with all the parts of your project as "submodules". That way, when you need to pull down some new code (but not any media assets) you can just do that, but everything is still being tracked by the superproject. Also, keep in mind that git will hash all of a commit at once. So if 500 2k files changed in a commit, the time spent hashing should be almost the same as hashing a 1MB file.
silentbicycle 18y ago
On several occasions, Perforce has introduced merge errors when merging from one branch to another.* Without mentioning (or speaking for) my employer, we have a project with about ten branches (4-5 in current use), 50-60k changelists in the history, 2-3 gb of data. The project is 10+ years old, but I don't believe the full history has been kept. We use Perforce, but several major developers don't fully trust it, and we're investigating other options. It does seem to handle large binary files reasonably well, at least, though fully scanning for any changes (the equivalent of "git status") generally takes about two minutes on my computer, so it's a mixed blessing. I think tracking binary data would be better handled by a fundamentally different kind of tool, really; there are major differences between managing large binaries vs. managing heuristically merge-able, predominantly textual data. * One example: Two functions with similar names but reversed arguments (i.e., methodA(from, to) and methodB(to, from)) had the arguments transposed during the merge in many, many files. It introduces some really subtle bugs. It also happened again during the next major merge from ongoing-development to release.
- TwoBit 18y ago
  You are telling Perforce to force-merge files without you having a say in it,and you are complaining that some of them didn't go right? Sorry, but you won't get better results from anything else if you have that policy.
jerf 18y ago
Some of your repliers are beating around the bush a little, I'm going to come right out and say it: Source control systems are for controlling source code. They are built from top to bottom around the idea that they are storing text files. They are build around the idea that a text-based patch is a meaningful thing to use. They are build around the idea that there is a reasonable merge algorithm to use to merge two people's changes. They can be used on things that physically resemble source code but aren't, like text-based document formats (raw HTML, for instance), but you probably won't need their full power and, by design, they lack features that would be useful in that case. For your convenience they are capable of storing small quantities of binary data since most projects have a little here and there. But in both cases, you're a bit out-of-spec. When you try to stuff tons of binary data into them, they break. They do not just break at the practical level, in the sense that operations take a long time but maybe someday somebody could fix them with enough performance work. They break at the conceptual level. Their whole worldview is no longer valid. The invariants they are built on are gone. It's not just a little problem, it's fundamental, and here's the really important thing: It can't be fixed without making them no longer great source control systems. I use git on a fairly big repository and the scanning is currently at the "annoying" level for me, but the scanning is there for good reasons, reasons related to its use as a source control system. On my small personal projects it definitely helps me a lot. SVN is the wrong tool for the job. I don't know what the right tool is. It may not even exist. But SVN is still the wrong tool. (If nothing else you could hack together one of those key/value stores that have been on HN lately and cobble together something with the resulting hash values.) And, going back to the original link, criticizing git for not working with a repository with large numbers of binary files is not a very interesting critique. If Perforce does work under those circumstances, I would conclude they've almost certainly had to make tradeoffs that make it a less powerful source control system. Based on what I've heard from Perforce users and critics, that is an accurate conclusion. But I have no direct experience myself.
- cousin_it 18y ago
  It can't be fixed without making them no longer great source control systems. Why? Is there some deep architectural reason why Git can't perform like Perforce on large binary files? Something so deep it cannot ever be fixed? I've read through this whole thread and see no such reason yet, only hints that it exists.
ajb 18y ago
There is an open source filesystem-based version control system: vesta (http://www.vestasys.org/ http://www.vestasys.org/). That's probably the sort of thing you need for this kind of work. It was developed by DEC and then Intel for chip development, and those guys check in binary blobs. Vesta is a cool piece of technology in many ways. It has a pure functional build language, completely parallelisable builds with accurate caching between builds, etc. The guy who maintains it at Intel is a bit bitter because he can't understand why less capable version control or build systems are more popular. Which is really because vesta's advantages show up best in quite large projects, but once your project is that large, it's very hard to switch. The web page looks moribund, but in fact it's still actively developed, and the developers hang out on IRC.

old-gregg 18y ago

We have a very similar problem [except we have videos that need to sit right next to the code tree] and we have added those video subfolders into .gitignore and use rsync for those instead - simple and effective tool-per-task. I don't believe in 6GB source trees without binary blobs. Perhaps Windows codebase is that large but I would assume it's hosted in multiple repos and very rarely is built/checked out by individual developers all at once.

silentbicycle 18y ago
> I don't believe in 6GB source trees without binary blobs. No kidding! Even the worst copy-and-paste programming would compress well. (Also, .gitignore + rsync seems like the best option to me, too.)

Semiapies 18y ago

Media assets are definitely an interesting problem. (Of course, then you're beyond "source control" and into something else, but that something else is important.)

windsurfer 18y ago

Git has a feature called "superprojects". You would configure this project with all the parts of your project as "submodules". That way, when you need to pull down some new code (but not any media assets) you can just do that, but everything is still being tracked by the superproject. Also, keep in mind that git will hash all of a commit at once. So if 500 2k files changed in a commit, the time spent hashing should be almost the same as hashing a 1MB file.

silentbicycle 18y ago

On several occasions, Perforce has introduced merge errors when merging from one branch to another.* Without mentioning (or speaking for) my employer, we have a project with about ten branches (4-5 in current use), 50-60k changelists in the history, 2-3 gb of data. The project is 10+ years old, but I don't believe the full history has been kept. We use Perforce, but several major developers don't fully trust it, and we're investigating other options. It does seem to handle large binary files reasonably well, at least, though fully scanning for any changes (the equivalent of "git status") generally takes about two minutes on my computer, so it's a mixed blessing. I think tracking binary data would be better handled by a fundamentally different kind of tool, really; there are major differences between managing large binaries vs. managing heuristically merge-able, predominantly textual data. * One example: Two functions with similar names but reversed arguments (i.e., methodA(from, to) and methodB(to, from)) had the arguments transposed during the merge in many, many files. It introduces some really subtle bugs. It also happened again during the next major merge from ongoing-development to release.

TwoBit 18y ago
You are telling Perforce to force-merge files without you having a say in it,and you are complaining that some of them didn't go right? Sorry, but you won't get better results from anything else if you have that policy.

jerf 18y ago

Some of your repliers are beating around the bush a little, I'm going to come right out and say it: Source control systems are for controlling source code. They are built from top to bottom around the idea that they are storing text files. They are build around the idea that a text-based patch is a meaningful thing to use. They are build around the idea that there is a reasonable merge algorithm to use to merge two people's changes. They can be used on things that physically resemble source code but aren't, like text-based document formats (raw HTML, for instance), but you probably won't need their full power and, by design, they lack features that would be useful in that case. For your convenience they are capable of storing small quantities of binary data since most projects have a little here and there. But in both cases, you're a bit out-of-spec. When you try to stuff tons of binary data into them, they break. They do not just break at the practical level, in the sense that operations take a long time but maybe someday somebody could fix them with enough performance work. They break at the conceptual level. Their whole worldview is no longer valid. The invariants they are built on are gone. It's not just a little problem, it's fundamental, and here's the really important thing: It can't be fixed without making them no longer great source control systems. I use git on a fairly big repository and the scanning is currently at the "annoying" level for me, but the scanning is there for good reasons, reasons related to its use as a source control system. On my small personal projects it definitely helps me a lot. SVN is the wrong tool for the job. I don't know what the right tool is. It may not even exist. But SVN is still the wrong tool. (If nothing else you could hack together one of those key/value stores that have been on HN lately and cobble together something with the resulting hash values.) And, going back to the original link, criticizing git for not working with a repository with large numbers of binary files is not a very interesting critique. If Perforce does work under those circumstances, I would conclude they've almost certainly had to make tradeoffs that make it a less powerful source control system. Based on what I've heard from Perforce users and critics, that is an accurate conclusion. But I have no direct experience myself.

cousin_it 18y ago
It can't be fixed without making them no longer great source control systems. Why? Is there some deep architectural reason why Git can't perform like Perforce on large binary files? Something so deep it cannot ever be fixed? I've read through this whole thread and see no such reason yet, only hints that it exists.

ajb 18y ago

There is an open source filesystem-based version control system: vesta (http://www.vestasys.org/ http://www.vestasys.org/). That's probably the sort of thing you need for this kind of work. It was developed by DEC and then Intel for chip development, and those guys check in binary blobs. Vesta is a cool piece of technology in many ways. It has a pure functional build language, completely parallelisable builds with accurate caching between builds, etc. The guy who maintains it at Intel is a bit bitter because he can't understand why less capable version control or build systems are more popular. Which is really because vesta's advantages show up best in quite large projects, but once your project is that large, it's very hard to switch. The web page looks moribund, but in fact it's still actively developed, and the developers hang out on IRC.