7 ms·
How do game companies share massive files?
- deleted 13y ago[deleted]
- t0 13y agoSneakernets are still the fastest way to transfer data. http://en.wikipedia.org/wiki/Sneakernet http://en.wikipedia.org/wiki/Sneakernet
- Piskvorrr 13y agoNot necessarily the cheapest, though - express shipping a bunch of disks is an uncertain venture ("oops, we accidentally dropped it a few times, is that bad?") and "I need a return cross-Atlantic flight ticket NOW" is sort of expensive. That said, I have heard of companies actually couriering semi-large volumes of sensitive data this way.
- Zenst 13y agoAlso in some area's there is a legal aspect due to law of some contract that makes sending somebody with a briefcase with data an easier way to move forward without triggering some technicality in contract or law. Indeed some countries and there data protection laws make things most interesting in legal ways to transfer data, even today. No standard on data protection World Wide and in itself a minefield, let alone other industries like medical research companies or those dealing with country wide firewalls.
- simias 13y agoSo basically rsync? Honestly synchronizing 50GB of contents across the globe doesn't sound that terribly novel or difficult to me, or the article makes a pretty bad job of explaining the difficulty.
- mryan 13y ago> That's because some of the complete game files were as large as 50GB 50GB is not the total size of the files needing to be transferred - some of the files were 50GB. Knowing the total size would have been interesting for the sake of discussion.
- debian69 13y agoYeah thats bollox , sorry but no game has 50 gb files on launch , and what kind of idiotic company would send round a file they weren't using in launch code base ?
- johnbellone 13y agoThe raw assets are generally a lot larger than what becomes present on your machine. An example would be normal mapping[1]. [1] http://en.wikipedia.org/wiki/Normal_mapping http://en.wikipedia.org/wiki/Normal_mapping
- lgbr 13y agoNot true anymore. http://www.explosion.com/45404/call-duty-ghosts-pc-install-50gb/ http://www.explosion.com/45404/call-duty-ghosts-pc-install-5...
- jswanson 13y agoTLDR; Internal workflows, not customer facing. Enterprise dropbox. The article title is misleading. They are actually talking about the internal workflow. Raw assets, art assets, and such, can be pretty large. When you have a studio in Country A doing PC, a studio in country B doing ps3, all wanting the base raw data but (presumably) transforming it in different ways, between how the build actually happens or artists tweaking something for a platform, it'd become a nightmare pretty quickly if you were to send it across a company private network (often just VPN pipes across the internet itself). The article is, at it's heart, an ad for a company called 'Panzura', which sounds like an enterprise-specific dropbox. Boxes 'on-premise', security guarantees, and probably pretty expensive.
- 13y ago
- golergka 13y agoSounds a lot like git or other typical solution.
- samnardoni 13y agogit would be pretty bad in this situation. It sounds more like rsync.
- golergka 13y agoThanks, I'll look into that.
- notduncansmith 13y agoNot gonna lie, when I saw "delta" I immediately assumed Git or some other source control solution. What boggles the mind is how these professional game developers recently discovered this technology. I didn't think Git was so exclusively beholden to the web development community.
- AndrewDucker 13y agoYou wouldn't use git for massive binary files that change on a regular basis. Particularly when you only need your testers to have the latest version of them, not the complete history (which is what git would give them).
- unwind 13y agoYou are fantastically mistaken if you think game devs don't use source control. Heh. Of course they do, and I'm ... let's say pretty sure that much of EA runs on http://www.perforce.com/ http://www.perforce.com/. Also weird that the article starts with a BF4 image, but then claims that BF4 was developed in Californa. I guess that hurts my national pride (go DICE!).
- NickPollard 13y agoGit solves a different problem; sending large binary blobs around is not it. As another comment said, this is essentially RSync, not git. I've been out of game development for a year, but unfortunately git is not being taken up much by large game developers. The main reason cited is actually git's handling of binary files, which leads to every checkout containing many duplicates of huge files. However this is possible to work around and the benefits of git still make it worth it.
- js2 13y agoThat said, git does have a delta compression mechanism for reducing disk space and saving network bandwidth that is quite complex: http://stackoverflow.com/questions/9478023/is-the-git-binary-diff-algorithm-delta-storage-standardized http://stackoverflow.com/questions/9478023/is-the-git-binary...
- deleted 13y ago[deleted]
- boothead 13y agoAs someone said on twitter the other day: If "tech experts" writing news articles aren't really experts, what does that say about all the other experts..?
- _sabe_ 13y agoSponsored article?
- JonnieCache 13y agoInnit. Someone at EA/Panzura's PR firm just got a raise.
- unsigner 13y agoAt some point Microsoft used a third-party solution called Aspera FASP to accept uploads of final game disc images. It was crazy fast - when you launched it, it reached our office' net connection speed within 10-15 sec, and other Internet connections at the office dropp.ed
- icelancer 13y agoI remember that! I worked at MS in one of the gaming capacities (no need to be specific) and it was unreal how close to maximum saturation FASP would get. Such an awesome tool.
- rurounijones 13y agoI have always wanted to know more about Valve's steam architecture. Why aren't they presenting at more conferences damnit!
- planetjones 13y agoof all the technical challenges facing us in the 21st century, sending a 50GB file quickly across some wire doesn't seem like the biggest! also the feedback i see about battlefield's stability and quality has been very poor (on ps4) so I had to laugh when I saw the " to locate defects and improve quality" quote :)
- moocowduckquack 13y agoAbove a certain point, it becomes far quicker to send a hard drive by courier.
- hengheng 13y agoThat used to be true, but considering that hard drive transfer rates cap out around 200 MByte/s, you'd have to send a large bunch of drives that must be read and written in parallel to be faster than a network connection.
- moocowduckquack 13y agoIf you are constantly mirroring your media drives with multiple redundancy, you can just unplug a set and post them.
- junto 13y agoSohonet 10Gbit/s. Used primarily by post production companies in Soho, London. http://www.sohonet.com/sohonet-media-network/ http://www.sohonet.com/sohonet-media-network/
- Zolomon 13y agoI think the article should mention that it is a Swedish based game developer company, and a California based publisher for the game. Sounds more fair that way.
- kayoone 13y ago"That's because some of the complete game files were as large as 50GB, and future games with more advanced graphics for the new Xbox One console and Sony's PlayStation 4 are likely to be even bigger." Ahm no, Battlefield 4 is already a PC game and scaled down even on the next gen consoles, so console processing power isnt the limiting factor here.
- deleted 13y ago[deleted]