11 ms·

The original version is apparently 45GB. I have no basis for saying this (I have not ever worked in the game industry), but my guess would be that this "patch"

• CydeWeys 8y ago
  It's always amazed me how downright inefficient patching is, even though better algorithms have existing for literally decades. The typical implementation is that you have to re-download every single resource that's changed, even if the resource is gigabytes large and only a bit in it has changed. Someone should calculate how many petabytes of Internet transfer capacity are wasted annually solely due to bad algorithms used in performing software updates. It'd be a surprisingly large figure.
  • toomuchtodo 8y ago
    Internet transfer capacity is not wasted unless it goes unused (or data transferred caused other data that was needed urgently to be queued). For example, you could say any links not currently near full utilization are going to waste.
    • egypturnash 8y ago
      If a hard drive doesn’t need to spin up because the patch you’re serving a zillion copies of fits in cache, you’re saving the power to run it and the heat it would have generated. Similarly with all the other resources that would be used on its journey between the data center and the end user. You’re wasting time if you could send an efficiently-generated 2 gig patch instead of a 60 gig file, especially for users in remote places. You’re taking up more space and cycles on your CDN. CPUs switch from idle power consumption to active. Every stage you can think of takes some amount of power and other resources, and if you’re delivering an unnecessarily huge file, you’re wasting everything it took to run all the steps along the way.
      • toomuchtodo 8y ago
        Human time is most valuable compared to the various waste you mention (cycles on CDNs, power, heat, etc).
        • CydeWeys 8y ago
          And a lot of human time is being wasted waiting for update operations that take much longer than necessary.
          • deleted 8y ago
            [deleted]
        • TeMPOraL 8y ago
          Yes, the oft repeated mantra in this industry. It's a cute thing to say when it's about saving your (developer) time by externalizing energy use on other people.
  • jayd16 8y ago
    Is there a way for public users to rsync(or whatever flavor of "better algoritm" you like) to traditional CDNs like Akamai?
    • toomuchtodo 8y ago
      The closest you'd come to such a setup is a squid proxy with some prefetching (besides bittorrent, which CDNs don't support until IPFS is a first class citizen).
    • kardos 8y ago
      The linux RPM world has "delta RPMs" which lets you ship much smaller packages when updating. I don't see any blocker to delivering these through a CDN. Not sure if this is quite what you were asking.
      • paulryanrogers 8y ago
        I believe the Deb world also has Jigsaw
    • asgeir 8y ago
      zchunk, zsync, and casync spring to mind. zchunk and zsync require support for range queries but casync splits the chunks into multiple files. https://github.com/zchunk/zchunk https://github.com/zchunk/zchunk http://zsync.moria.org.uk/ http://zsync.moria.org.uk/ https://github.com/systemd/casync https://github.com/systemd/casync
  • kardos 8y ago
    > It'd be a surprisingly large figure. It would be a rather large figure, but I'd wager that superfluous video streaming (eg, video advertising, or youtube/netflix autoplaying to no audience) overwhelmingly dwarfs any bandwidth wasted on suboptimal update delivery.
  • Jasper_ 8y ago
    SteamPipe breaks all bits of content up into ~1MiB chunks, and then fetches those chunks. As long as you align your packfiles to 1MB boundaries, and keep your content sorted, only new chunks will be downloaded. Unfortunately, a lot of developers don't follow these guidelines for a number of reasons, and basically generate a whole new set of chunks with every update. This includes a game I work on, because it's built in a version of Unreal where we do not want to drastically change the packfile format and modify the build process to handle this case.
    • PezzaDev 8y ago
      Do you know where I could find details about how to optimise for SteamPipe?
      • Jasper_ 8y ago
        https://partner.steamgames.com/doc/sdk/uploading#Building_Efficient_Depots https://partner.steamgames.com/doc/sdk/uploading#Building_Ef...
  • deleted 8y ago
    [deleted]
  • pcwalton 8y ago
    One important issue is that if the patch program goes wrong, you can easily end up with customers having broken installations that you can't fix. The patching software is how you distribute updates, so it has to work. So there's strong incentive to deploy simple techniques with fewer failure modes as opposed to optimizing away every last byte of network traffic. Firefox has historically shied away from complicated patch algorithms for this reason among others. (To be fair, of course, we're also dealing with several orders of magnitude less data than AAA games are...)
    • deleted 8y ago
      [deleted]
    • CydeWeys 8y ago
      If the patch doesn't apply cleanly, don't apply it. The command-line `patch` utility works just fine for this. Transactional updates with rollback on failure are a solved problem.
      • kbenson 8y ago
        Is patch generally applied to resources I the multiple gigabyte range, very possibly with the original resource size exceeding the total amount of installed RAM in the system? Patching small to moderate sized resources transactional ly might be a solved problem, but I imagine doing so on large resources in an acceptable time frame may not be for this use case.
        • TeMPOraL 8y ago
          > Is patch generally applied to resources I the multiple gigabyte range, very possibly with the original resource size exceeding the total amount of installed RAM in the system? That's where bundling all your data files into a single large archive bites you in the butt. Individual files the game uses ain't gonna be that large.
          • Tehdasi 8y ago
  • TrevorJ 8y ago
    I've been pretty surprised at how good a job steam does at this. When I release an update for my game, the download size is typically quite tiny relative to all the files I've touched.
    • hamilyon2 8y ago
      Steam itself however downloads full installation, as it seems. Which is interesting.
      • gmueckl 8y ago
        Self-updating on Windows is a horrible thing to get right. I wouldn't be surprised if they did this to keep the complexity of the process as low as possible.
      • makomk 8y ago
        If a game update goes wrong, they can just send down another update. If a Steam update goes wrong... (This has sort of happened. Valve managed to roll out a version of the launcher for Linux a while back which used instructions not available on older CPUs, and crashed before it reached the point where it could check for updates. Was a pain. Normally, if Steam exits uncleanly they force an update check immediately on next restart, but if it never gets that far you're screwed.)
  • pizzapill 8y ago
    Only companies that are bad at tech implement it this way. Bethesda is really bad at tech. They use a ancient engine that binds physics calculation to frames and can't go over 60 FPS. They warned Fallout 76 fans during the Beta that they should not click the "Update" button (big download, game screwed). They trust the game client 100%. You can hack/cheat in this multiplayer game by changing INI files with a text editor.
  • skunkworker 8y ago
    Apple has had delta app updating since iOS 6 and I know that Steam also does this. The real issue becomes a combination of, 1. How fast is your internet connection. 2. How fast is your HDD to read the newly downloaded data and patch the existing file. (When I used to have my steam games/windows on an HDD this took forever.)
  • zorked 8y ago
    Among others there's the issue that patching may be a seek-heavy operation. Downloading a 40GB file over a fiber connection is much faster than merging 8GB of new textures into a 32G file on a console that has a HDD. When downloading from a fiber connection to a PS4 HDD the bottleneck is likely the HDD's speed. To merge a file you would need to at least read it and write it again, which is at least twice as slow as simply downloading it. Alternatively you can pay the cost by seeking constantly instead (like a texture per file), which may be even slower and probably sucks at runtime.
    • mejari 8y ago
      Only a quarter of the US even has the option of fiber, much less actually have it. It's silly to use ideal fiber transfer rates as the argument for downloading whole-hog vs more targeted patching, and I doubt it's the reasoning they use.
      • Tsubasachan 8y ago
        And don't forget that Sony and Microsoft need the infrastructure in place for handling millions of people patching the game at the same time...
    • tracker1 8y ago
      If it's one 32G file, there's already something wrong...
  • billfruit 8y ago
    The pirates seems to have mastered the art of compressing assets to create distribution packages of manageable size. Maybe legit publishers needs to learn a few lessons from them.
    • Sohcahtoa82 8y ago
      There was one game (I think one of the really old Unreal Tournament games) that had all the sound files as .wav files, which made the game huge. Many people were still on dialup at the time, so pirating it would have taken over a week to download it. Well, one pirate group compressed all the .wav files into MP3s and wrote a quick batch script to decompress them all back into the .wav's. Turned a 700 megabyte download into a little under 100 megabytes.
      • nullify88 8y ago
        Jeez that takes me back. CORE, CLS... Some games were compressed with UHARC or ACE which performed better than other compressors at the time. mp3unpack or WAVE injector would also run to decompress the mp3s or recombine them back in to large solid files. All that while having some banging chip tune and a snazzy animation play. CLASS!
      • lacker 8y ago
        At that time it might also have been illegal to distribute an mp3 decoder without paying royalties.
    • krferriter 8y ago
      That's because pirates care more about user experience and know more about digital packaging than the publishers do. People getting content from pirates can just switch to another source at any time if one pirate does a bad job. People getting content from official publishers are stuck with whatever they put out.
  • mjevans 8y ago
    Basic idea: If the file is smaller than 10x the base patch unit size: don't bother just add the whole file as a single unit / send it all for each update. Else: Create a manifest of the file: Size on Disk, Checksum of file, Name of file mapping to (one or more) list of segments within file by segment offset, segment size, segment checksum. A custom list might be provided by a resource packer that is aware of assets / code segments within the file. A website will provide an interface that has a list of file names to target size/hashes. There will also be a list by size/hash that enumerates the segment lists within the file. To update, the file size/checksum would be computed and the manifests obtained from the server; these would be compared to the files on disk and invalid segments/files discarded. The target version would then be selected and the target manifests collected. First limited by filename and then optionally project wide, segments would be matched first checking size then checksum. There are a couple different methods that might then be optimal depending on the desired outcome, which depend on disk space, operating system support for hard-linking/de-duplicating subsegments of a file, if the media is known to be an SSD or spinning rust, etc. However, in any event, that infrastructure allows for easily checking and upgrading / repairing to any version from any version at the expense of storing a small quantity of metadata and having full copies via the actual delivery service for segments on file. The backend might also have de-duplication of the stored data as a bonus. This is probably a very naive implementation of compression via de-duplication and should be obvious to anyone skilled in this form of art.
• makomk 8y ago
  More like the complete version happens to be 50 GB larger than the shipped on-disc data, judging from the 96 GB install size. The maximum size of a Blu-Ray is about 50 GB, so we're hitting the point where games are too big to even fit on a single disc and it's almost just acting as a dongle for the download version.
  • nikofeyn 8y ago
    red dead redemption 2 recently released and required two discs. one was data only, whcih was copied off. the other disc was data plus the playable disc. this is the first ps4 game i have seen do that, but i don't know if there are others.
    • jvanderbot 8y ago
      Oh man this brings me back to 5 CD jrpgs. We had to know it was inevitable.
      • aidenn0 8y ago
        I think Wing Commander IV was the most discs of any game I owned (6 IIRC). Either Wing Commander II or Gabriel Knight was the most floppies (both around 12).
      • gmassman 8y ago
        Yeah iirc Call of Duty 2 had 5 discs on PC
        • selectodude 8y ago
          Unreal Tournament 2004 was like 7 CDs. I remember having to buy a second hard drive for that game. Worth it.
      • SXX 8y ago
        If I remember correctly original Phantasmagoria release that I still have had like 7 discs.
• hacker_9 8y ago
  11gb isn't some bug fix, this is asset overhauls. Be it models, texture, audio etc. This is a result of Bethesda overworking employees and crunching like there's no tomorrow. The reason for the download being this size is because they aren't technically capable of making it smaller, and just wrongly assume everyone has the network capability to download it instead. It would take 3 days on my non-fibre internet to download though, so this is a purchase I won't be making.
  • katamaritaco 8y ago
    It's an online only game. If this would really take you 3 days to download, you wouldn't be able to play it anyways...
    • Rychard 8y ago
      > It's an online only game. If this would really take you 3 days to download, you wouldn't be able to play it anyways... (54 GB) / (3 days) = 1.66666667 megabits per second To really drive this point home, even the FCC agrees with the above poster: https://www.fcc.gov/reports-research/guides/broadband-speed-guide https://www.fcc.gov/reports-research/guides/broadband-speed-...