4 ms·

That's really cool. There are a bunch of tools that will let you symlink or hard link deduplicate files, but being able to do block-level dedupes simply by lean

• mikepurvis 4y ago
  Yes, Steamworks encourages developers to align assets to certain boundaries to help ensure efficient patching later, see the documentation here: https://partner.steamgames.com/doc/sdk/uploading https://partner.steamgames.com/doc/sdk/uploading
  • mastax 4y ago
    Interesting, it looks like steam doesn't have any special handling for inserts/offsets and just handles 1MB blocks. The extra disk IO probably just comes from rewriting large files: > Next - to update the pack file on a client device, SteamPipe builds the new version alongside the old version. When all new files are built, it then "commits" the update by deleting old files and moving the new files in. What this means is that to update a 25 GB pack file, SteamPipe will always build a new, 25GB file. If the update only requires 10 bytes of changes to that file, SteamPipe will need to copy almost the entire 25GB from the old file to the new file. Depending on the client storage hardware, this can be a very slow process.
    • bob1029 4y ago
      I think SQLite + WAL could be a compelling solution for game patching scenario. There are already SQLite clustering technologies that utilize WAL frame shipping in order to replicate data. I don't see why a steam patch couldn't just be a range of WAL frames to be applied to a client's database.
      • ilyt 4y ago
        I think that train already sailed. With HDD or SDD that might've been feasible but with NVMe you really want as little between raw storage and game engine.
        • mastax 4y ago
          Yeah, it would probably be easier to move back to individual files from packfiles. Assuming that the main motivation for packfiles was layout for HDDs and not filesystem overhead.
          • ilyt 4y ago
            Initially packfiles were absolutely about filesystem overhead, "a lot of small files" were terrible performance-wise. OSes got better tho, don't know in windows side, but on linux side "few tens of thousands of small files" case really isn't that bad But I'm really talking about ability to just directly load an asse via reading a blob of data vs having to munge it thru SQLite code. Or even just mmap'ing data directly
            • account42 4y ago
              Windows is still extremely painful with many small files (at last the Win32 API, maybe going directly via Nt* functions is slightly better) and it only gets worse once you add "Antivirus" Malware that wants to hook into all FS access.
      • mikepurvis 4y ago
        Yes, but I think this workflow is dependent on an assumption that there’s no cost overhead for random accesses, which only became true in the current console generation, with mandatory fast SSDs. As long as you were potentially seeking around a spinning platter, you needed to have way more control over data locality than was possible with something like SQLite.
• beecafe 4y ago
  What about using content defined chunking? Then inserting a few lines shouldn't cause all the subsequent chunks to shift
  • viraptor 4y ago
    That's not quite right. The contents in the file would still shift. You'd just be able to still deduplicate most chunks between multiple files... But only in your copies because the filesystem still uses constant size chunks.
    • riceart 4y ago
      But I think they’re talking about if you’re doing content defined chunking of a pack or archive file - inserting data should only affect insertion chunks + 1. And since it’s content defined - those chunks are necessarily not constant size. Ie this is how backup tools like Arq or duplicacy work.
      • saltcured 4y ago
        Backup tools don't necessarily model mutation, since their purpose is to encode immutable snapshots of data. The modern ones that use deduped content storage also generally like to make snapshots independent of one another, so you can prune snapshots without worrying about invalidating another dependent snapshot. As a result, introducing a new version of a file likely has the same O(n) cost as backing up a new file, as long as they both contain content chunks already found in the storage area. These backup tools also don't generally have to optimize random access to parts of a file. They may just store a linear sequence of chunk IDs to represent one file version. To bring this back to an active system that supports random access by a program with patching, I think you'd really need to adapt a copy-on-write filesystem to use content-defined chunking instead of fixed offset chunks. Then, your insertion is likely an operation on some tree structure that represents the list of chunk IDs as the leaves of the tree. But, this tree would now have to encode more byte offset info in the interior tree nodes, since it would vary at the leaves instead of each leaf representing a fixed size chunk.
• panzi 4y ago
  Some games just add an update pak that overloads the changed files, not changing the original pak file. So the installation size of such games might grow a lot on updates, wasting a lot of space by keeping the old files, but the files are immutable at least and rollbacks of updates are theoretically possible.
• account42 4y ago
  Steam resisted any patching mechanism for updates for a long time. Even when they revamped the update system [0] they instead opted to split updates into separate files for their own pack format [1]. They seem to have relented though - I guess other developers were not as motivated to optimize their asset layout for Steam when Valve is the one paying for the bandwidth. [0] https://developer.valvesoftware.com/wiki/SteamPipe https://developer.valvesoftware.com/wiki/SteamPipe [1] https://developer.valvesoftware.com/wiki/VPK https://developer.valvesoftware.com/wiki/VPK