4 ms·

Game development. Setting up a new workspace generally is a "go out to lunch and it'll hopefully be mostly done with the perforce checkout by the time you're b

Game development. Setting up a new workspace generally is a "go out to lunch and it'll hopefully be mostly done with the perforce checkout by the time you're back" affair on our internal LAN, nevermind if you wanted to work from home. Downloading the built assets through our internal tools takes a similar length of time, so you hopefully can wait until the end of the day to kick that off as you head out. Various SDKs are delivered by download - my last download was 15GB, compressed. These often require frequent updates, which we save to local shares as a simple matter of expediency when shared widely. Delivering complete, self contained milestones in specific formats causing 12-50GB .zips to be uploaded via FTP for hours... with the semi-frequent resort to sneakernet due to quota limitations, service issues, or just plain not having hours between when development for a build was stopped and when the build was needed someplace for a demo or whatever else. I have an archive of some of those deliveries on my local machine. Even with a good 30%+ of my archives missing due to lack of drive space, the archives for the single project I've archives for total over a terrabyte. They're useful for reproducing bugs exactly on occasion. Between all the other projects, other employees, and monstrous shared resources like complete Perforce history, build machines and other servers... complete backup would probably be somewhere around the 0.1 petabytes mark... or close to a full petabit of data. And we're considered only a "small" business, under 50 employees! Fortunately for us backups are fairly static, and the nightly rsync to offsite storage is usually done bringing all our services to a grinding halt by the time work starts the next day. But it's sufficiently slow that those are our secondary backups, with primaries and restoration being done through sneakernet as the rule, not the exception, despite the "beefy" pipe. A second internet backup would push backup times into our workday, and if we're ever forced to do a major restoration through the internet, it's going to be extremely painful... days of waiting with employees unable to do their jobs. Expensive!

chubot 13y ago
Wow I'm surprised game development is still like that. I haven't done game development in almost 9 years, but the "Perforce saturating the LAN" problem sounds very familiar! So I'm actually working on a (binary) software distribution system, and I think it would work very well for this use case. This problem can be solved with software -- it doesn't require an upgrade to fiber. The (straightforward) idea is to use content-addressed storage and differential compression, pretty much like Git does. Then you can actually sync data from your neighbor's machine, in a BitTorrent-ish fashion (possibly getting tiny amounts of metadata from a central server for consistency). Perforce definitely has problems with spurious lock contention, and just a few team members syncing from the same machine can easily clog its pipes if you're not careful. I'm not up to date, but I'm pretty sure you can do better than Perforce's delta compression. You can probably do it with a single generic algorithm, but one way to really improve would be to use file-type specific compression, e.g. bsdiff or Chrome's Courgette for executables, and use other heuristics for game data, raw audio, video, etc. You actually want to avoid single-file compression in the repository so you can take advantage of the differential compression. I can understand that the dynamics of game development teams means that this will never get written in house (although perhaps in the years since I've left, people started placing more emphasis on tools). But for all the entrepreneurs out there, I wouldn't dismiss the market of selling tools to game developers. Perforce is a tiny company that I know made an absolute killing in that market, and I'm somewhat surprised that after a decade they're still the state of the art.
- MaulingMonkey 13y ago
  > Wow I'm surprised game development is still like that. Some of the problems are fundamental. We're forced to use certain compressed and signed packaging schemes which aren't sanely delta compressable. We distribute in self contained blobs because not everyone has older versions lying around, or will not have a worthwhile pipe when they get to where they need to install things and will further sneakernet on their end. Even if we did assume sane deltas are possible, simple information density puts a lower bound on patch size based on how much content changed, and even that theoretical lower bound can frequently be too much and take too long for relatively "beefy" pipes. Heck, even with fiber, people will still be saving these things out to USB drives for good reason. > Then you can actually sync data from your neighbor's machine, in a BitTorrent-ish fashion (possibly getting tiny amounts of metadata from a central server for consistency). Perforce definitely has problems with spurious lock contention, and just a few team members syncing from the same machine can easily clog its pipes if you're not careful. That helps if the bottleneck is the server. Our perforce machines are beefy enough that our bottlenecks are frequently the client and the LAN pipe, neither of which will be helped by BitTorrent style networking. Work from home would be 100% pipe bottlenecked, as the work pipe is able to saturate it quite easily. FTP is still bottlenecked by either the developer's or the publisher's pipe, and wouldn't be helped by p2p either. > I'm not up to date, but I'm pretty sure you can do better than Perforce's delta compression. There's always room for incremental improvements... but not enough to beat out fiber. > You actually want to avoid single-file compression in the repository so you can take advantage of the differential compression. Not always possible due to format requirements. Windows 8 .appx packages, for example, are basically self contained signed .zips containing an application and all it's resources. Want to provide three builds? You just compressed and signed three different entire copies of all your built resources. For bonus points, those were .zip ed again. Insanity! But software cannot magically solve the social problems that let such designs reach the marketplace. And since things break unless you use the exact "correct" certificate for signing, any script that generated the .appx from one set of resources would require distributing the actual private key which is a non starter...

chubot 13y ago

Wow I'm surprised game development is still like that. I haven't done game development in almost 9 years, but the "Perforce saturating the LAN" problem sounds very familiar! So I'm actually working on a (binary) software distribution system, and I think it would work very well for this use case. This problem can be solved with software -- it doesn't require an upgrade to fiber. The (straightforward) idea is to use content-addressed storage and differential compression, pretty much like Git does. Then you can actually sync data from your neighbor's machine, in a BitTorrent-ish fashion (possibly getting tiny amounts of metadata from a central server for consistency). Perforce definitely has problems with spurious lock contention, and just a few team members syncing from the same machine can easily clog its pipes if you're not careful. I'm not up to date, but I'm pretty sure you can do better than Perforce's delta compression. You can probably do it with a single generic algorithm, but one way to really improve would be to use file-type specific compression, e.g. bsdiff or Chrome's Courgette for executables, and use other heuristics for game data, raw audio, video, etc. You actually want to avoid single-file compression in the repository so you can take advantage of the differential compression. I can understand that the dynamics of game development teams means that this will never get written in house (although perhaps in the years since I've left, people started placing more emphasis on tools). But for all the entrepreneurs out there, I wouldn't dismiss the market of selling tools to game developers. Perforce is a tiny company that I know made an absolute killing in that market, and I'm somewhat surprised that after a decade they're still the state of the art.

MaulingMonkey 13y ago
> Wow I'm surprised game development is still like that. Some of the problems are fundamental. We're forced to use certain compressed and signed packaging schemes which aren't sanely delta compressable. We distribute in self contained blobs because not everyone has older versions lying around, or will not have a worthwhile pipe when they get to where they need to install things and will further sneakernet on their end. Even if we did assume sane deltas are possible, simple information density puts a lower bound on patch size based on how much content changed, and even that theoretical lower bound can frequently be too much and take too long for relatively "beefy" pipes. Heck, even with fiber, people will still be saving these things out to USB drives for good reason. > Then you can actually sync data from your neighbor's machine, in a BitTorrent-ish fashion (possibly getting tiny amounts of metadata from a central server for consistency). Perforce definitely has problems with spurious lock contention, and just a few team members syncing from the same machine can easily clog its pipes if you're not careful. That helps if the bottleneck is the server. Our perforce machines are beefy enough that our bottlenecks are frequently the client and the LAN pipe, neither of which will be helped by BitTorrent style networking. Work from home would be 100% pipe bottlenecked, as the work pipe is able to saturate it quite easily. FTP is still bottlenecked by either the developer's or the publisher's pipe, and wouldn't be helped by p2p either. > I'm not up to date, but I'm pretty sure you can do better than Perforce's delta compression. There's always room for incremental improvements... but not enough to beat out fiber. > You actually want to avoid single-file compression in the repository so you can take advantage of the differential compression. Not always possible due to format requirements. Windows 8 .appx packages, for example, are basically self contained signed .zips containing an application and all it's resources. Want to provide three builds? You just compressed and signed three different entire copies of all your built resources. For bonus points, those were .zip ed again. Insanity! But software cannot magically solve the social problems that let such designs reach the marketplace. And since things break unless you use the exact "correct" certificate for signing, any script that generated the .appx from one set of resources would require distributing the actual private key which is a non starter...