4 ms·

Interesting, it looks like steam doesn't have any special handling for inserts/offsets and just handles 1MB blocks. The extra disk IO probably just comes from r

by mastax 4y ago

Interesting, it looks like steam doesn't have any special handling for inserts/offsets and just handles 1MB blocks. The extra disk IO probably just comes from rewriting large files: > Next - to update the pack file on a client device, SteamPipe builds the new version alongside the old version. When all new files are built, it then "commits" the update by deleting old files and moving the new files in. What this means is that to update a 25 GB pack file, SteamPipe will always build a new, 25GB file. If the update only requires 10 bytes of changes to that file, SteamPipe will need to copy almost the entire 25GB from the old file to the new file. Depending on the client storage hardware, this can be a very slow process.

bob1029 4y ago
I think SQLite + WAL could be a compelling solution for game patching scenario. There are already SQLite clustering technologies that utilize WAL frame shipping in order to replicate data. I don't see why a steam patch couldn't just be a range of WAL frames to be applied to a client's database.
- ilyt 4y ago
  I think that train already sailed. With HDD or SDD that might've been feasible but with NVMe you really want as little between raw storage and game engine.
  - mastax 4y ago
    Yeah, it would probably be easier to move back to individual files from packfiles. Assuming that the main motivation for packfiles was layout for HDDs and not filesystem overhead.
    - ilyt 4y ago
      Initially packfiles were absolutely about filesystem overhead, "a lot of small files" were terrible performance-wise. OSes got better tho, don't know in windows side, but on linux side "few tens of thousands of small files" case really isn't that bad But I'm really talking about ability to just directly load an asse via reading a blob of data vs having to munge it thru SQLite code. Or even just mmap'ing data directly
      - account42 4y ago
        Windows is still extremely painful with many small files (at last the Win32 API, maybe going directly via Nt* functions is slightly better) and it only gets worse once you add "Antivirus" Malware that wants to hook into all FS access.
- mikepurvis 4y ago
  Yes, but I think this workflow is dependent on an assumption that there’s no cost overhead for random accesses, which only became true in the current console generation, with mandatory fast SSDs. As long as you were potentially seeking around a spinning platter, you needed to have way more control over data locality than was possible with something like SQLite.
- justinclift 4y ago
  > There are already SQLite clustering technologies that utilize WAL frame shipping in order to replicate data. Would you be ok to point out ones that do this? Searching online isn't showing up anything mentioning WAL frame shipping for SQLite. :( Was thinking about exactly this scenario a few days ago, as I have a potential use for it. :)
  - justinclift 4y ago
    Ahhh, at least Litestream seems to do this: https://litestream.io/how-it-works/ Awesome. :)
- gcr 4y ago
  interesting, is this a supported SQLite feature? I understand that when the WAL file gets too big (e.g. 4mb), SQLite will then checkpoint its contents back to the main sqlite database. How can one generate a WAL file big enough to contain a game patch?
garaetjjte 4y ago
As far I can tell Steam does have two mechanisms. First is chunk based matching using rolling checksums, which is something like zsync/precomputed rsync. This allows to switch between arbitrary build manifests and resuses matching chunks from previous file. Second is based on computing patches with smaller granularity, but generation of these is only triggered when enough clients download the update. They are more efficient but valid only for updating between exact build manifest pair they were generated for. When compressed packfiles are used they should be built with equivalent of gzip --rsyncable option, which restarts compression stream based on rolling hash values. Otherwise patching methods are defeated. For example UnityFS packfiles with chunked LZ4 compression normally don't do that, which results in huge updates. However it is possible to repack them into patch friendly form while keeping file format compatibility.
LorenPechtel 4y ago
If there is any compression going on you don't really have a choice. Long ago I have actually done code that updated files (one field per record, no size changes) in a .zip file but the creation specifically forced the files in question to be stored with no compression. (Legacy evolution--all the files of a job were packed into a single .zip. Then a need came along to update some of them. Nothing else would even notice that the files hadn't been compressed, but the one task could very quickly write the changes without replacing the file. Update the relevant byte, recalc the CRC from the in-memory copy, write it to both locations.)

by mastax 4y ago

bob1029 4y ago

I think SQLite + WAL could be a compelling solution for game patching scenario. There are already SQLite clustering technologies that utilize WAL frame shipping in order to replicate data. I don't see why a steam patch couldn't just be a range of WAL frames to be applied to a client's database.

ilyt 4y ago
I think that train already sailed. With HDD or SDD that might've been feasible but with NVMe you really want as little between raw storage and game engine.
- mastax 4y ago
  Yeah, it would probably be easier to move back to individual files from packfiles. Assuming that the main motivation for packfiles was layout for HDDs and not filesystem overhead.
  - ilyt 4y ago
    Initially packfiles were absolutely about filesystem overhead, "a lot of small files" were terrible performance-wise. OSes got better tho, don't know in windows side, but on linux side "few tens of thousands of small files" case really isn't that bad But I'm really talking about ability to just directly load an asse via reading a blob of data vs having to munge it thru SQLite code. Or even just mmap'ing data directly
    - account42 4y ago
      Windows is still extremely painful with many small files (at last the Win32 API, maybe going directly via Nt* functions is slightly better) and it only gets worse once you add "Antivirus" Malware that wants to hook into all FS access.
mikepurvis 4y ago
Yes, but I think this workflow is dependent on an assumption that there’s no cost overhead for random accesses, which only became true in the current console generation, with mandatory fast SSDs. As long as you were potentially seeking around a spinning platter, you needed to have way more control over data locality than was possible with something like SQLite.
justinclift 4y ago
> There are already SQLite clustering technologies that utilize WAL frame shipping in order to replicate data. Would you be ok to point out ones that do this? Searching online isn't showing up anything mentioning WAL frame shipping for SQLite. :( Was thinking about exactly this scenario a few days ago, as I have a potential use for it. :)
- justinclift 4y ago
  Ahhh, at least Litestream seems to do this: https://litestream.io/how-it-works/ Awesome. :)
gcr 4y ago
interesting, is this a supported SQLite feature? I understand that when the WAL file gets too big (e.g. 4mb), SQLite will then checkpoint its contents back to the main sqlite database. How can one generate a WAL file big enough to contain a game patch?

garaetjjte 4y ago

As far I can tell Steam does have two mechanisms. First is chunk based matching using rolling checksums, which is something like zsync/precomputed rsync. This allows to switch between arbitrary build manifests and resuses matching chunks from previous file. Second is based on computing patches with smaller granularity, but generation of these is only triggered when enough clients download the update. They are more efficient but valid only for updating between exact build manifest pair they were generated for. When compressed packfiles are used they should be built with equivalent of gzip --rsyncable option, which restarts compression stream based on rolling hash values. Otherwise patching methods are defeated. For example UnityFS packfiles with chunked LZ4 compression normally don't do that, which results in huge updates. However it is possible to repack them into patch friendly form while keeping file format compatibility.

LorenPechtel 4y ago

If there is any compression going on you don't really have a choice. Long ago I have actually done code that updated files (one field per record, no size changes) in a .zip file but the creation specifically forced the files in question to be stored with no compression. (Legacy evolution--all the files of a job were packed into a single .zip. Then a need came along to update some of them. Nothing else would even notice that the files hadn't been compressed, but the one task could very quickly write the changes without replacing the file. Update the relevant byte, recalc the CRC from the in-memory copy, write it to both locations.)