9 ms·

Craziest thing I ever used SQLite for: partial file deduplication (2022)

343 points by ics 4y ago

sgtnoodle 4y ago
That's neat! In retrospect, did the database make the problem more tractable from an algorithmic or practical standpoint, or was it mostly just using the tools you're familiar with? If I were to approach the same problem, I likely would have tried to keep all the data in RAM and serialize it out to a file for incremental runs.
- qsort 4y ago
  IME a database does not algorithmically improve anything, but it does make at your disposal some very efficient data structures that are definitely nontrivial to replicate. This is especially the case if you have to work with datasets that don't fit in memory! An added bonus is that you can use SQL.
  - zerkten 4y ago
    A friend works for a financial company that gets applicants from .NET and Linux backgrounds. He observed that the .NET devs almost always thought about solving these problems with a database and SQL that the Linux people would solve with pipelines of commands. Both were valid in his environment since they were both a big Oracle and SQL Server shop while having high Linux usage as well. I must ask him if PowerShell has changed things at all.
- theamk 4y ago
  Definitely the latter. There are many of other products which implement "keep track of partial file contents to deduplicate", for example casync borg and restic. None of them use sqlite for metadata storage, which kinda makes sense - in author's schema, each block uses (fileid, block_index, hash) tuple, which is 2/3 wasted space, as fileids are mostly same (if files are big) and block_index is always increasing. A custom data structure would be both faster and more efficient.
- nonethewiser 4y ago
  I have used SQLite to deduplicate csvs when it couldn’t all fit into memory. I imagine Dask would be a better tool for this job but I wasn’t familiar with it at the time and it did allow me to deduplicate using Python and no external dependencies.
hultner 4y ago
Very cool and probably a fun exercise, but I would probably put the data on a ZFS volume with dedupe instead, which from reading this implementation is pretty much the same thing to my layman’s perspective. I could also add compression to the same dataset.
- traceroute66 4y ago
  > I would probably put the data on a ZFS volume with dedupe instead But doesn't ZFS dedupe eat RAM for breakfast ? Double-digit GB RAM per TB data IIRC ?
  - giantrobot 4y ago
    It's usually along the lines of 1GB per TB. Some factors can affect that number but I've found it pretty accurate. Note that's 1GB of RAM that ends up wired to ZFS and never usable by the rest of the system.
  - redundantly 4y ago
    The DDT (deduplication table) used to require ~1GB RAM for every 1TB data written over the life of the file system. Deleting data from the file system wouldn't remove the dedupe references, you'd have to recreate the pool entirely and start over with a new DDT. However, there are now special devices that can be used stored to store DDT. Typically this is done with two SSDs configured as a mirrored vdev for the DDT metadata. This reduces the overhead on memory, but does cost some performance and still has the same limitation that the DDT size can only be reduced by re-creating the pool.
    - hultner 4y ago
      Yes this is what I do, and I wouldn't have dedupe for all datasets, for instance in this case he wanted dedupe on his games, so I would just enable it for that dataset. But I do run disk-wide compression with no problems and have done so on all my datasets for many years now, and it's been a tremendous space saver. Especially on machines where I have a lot of VMs/containers, it's not unusual for me to have a compression ratio of 2 on these with good old lz4, it will be interesting to see what damage zstd will do once I start experimenting with that.
evmar 4y ago
Coincidentally, I just saw that the "USENIX Test of Time Award" for 2023 was an analysis for deduplication purposes of real-world files. I found Figure 1 particularly interesting in that partial deduplication didn't save much over whole-file based deduplication in practice. https://www.usenix.org/conferences/test-of-time-awards https://www.usenix.org/conferences/test-of-time-awards
- emmelaich 4y ago
  You'd expect that in general, but for developers updating content the partial would often win.
mastax 4y ago
That's really cool. There are a bunch of tools that will let you symlink or hard link deduplicate files, but being able to do block-level dedupes simply by leaning on the filesystem is nice. It sometimes feels like games are made to thwart this type of thing. They often use packfiles, basically filesystems within files optimized to look up assets quickly. Also perhaps they allowed optimized data layout from when consoles had slow spinning hard drives. The upshot is that a tiny patch inserting a line of code in a script may offset hundreds of megabytes of other data in the packfiles, causing the block hashes to no longer match up. Do any filesystems model inserts in some way? I'm pretty sure Steam updates can handle situations like that. I frequently see updates which download a tiny amount (kilobytes) but write a huge amount to disk (gigabytes), and I can't think of any other cause. (Assuming developers aren't using hilariously un-compressed assets).
- mikepurvis 4y ago
  Yes, Steamworks encourages developers to align assets to certain boundaries to help ensure efficient patching later, see the documentation here: https://partner.steamgames.com/doc/sdk/uploading https://partner.steamgames.com/doc/sdk/uploading
  - mastax 4y ago
    Interesting, it looks like steam doesn't have any special handling for inserts/offsets and just handles 1MB blocks. The extra disk IO probably just comes from rewriting large files: > Next - to update the pack file on a client device, SteamPipe builds the new version alongside the old version. When all new files are built, it then "commits" the update by deleting old files and moving the new files in. What this means is that to update a 25 GB pack file, SteamPipe will always build a new, 25GB file. If the update only requires 10 bytes of changes to that file, SteamPipe will need to copy almost the entire 25GB from the old file to the new file. Depending on the client storage hardware, this can be a very slow process.
    - bob1029 4y ago
      I think SQLite + WAL could be a compelling solution for game patching scenario. There are already SQLite clustering technologies that utilize WAL frame shipping in order to replicate data. I don't see why a steam patch couldn't just be a range of WAL frames to be applied to a client's database.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
rurban 4y ago
Using proper a hash table with a bloom filter would save you the useless pass through a b-tree though. Much faster and much less memory
- pornel 4y ago
  With a b-tree you can do even better. Instead of hashing entire files ahead of time, you can avoid hashing more than the minimum prefix required to conclude a file is unique, by building the b-tree lazily. I don't think it can be done with sqlite though, as it requires hashing the files while traversing the b-tree. https://lib.rs/crates/dupe-krill#readme-nerding-out-about-the-fast-deduplication-algorithm https://lib.rs/crates/dupe-krill#readme-nerding-out-about-th...
JackSlateur 4y ago
Looks like https://github.com/markfasheh/duperemove https://github.com/markfasheh/duperemove
- philsnow 4y ago
  Oh, hey Mark (Not (purely) a reference to the “oh hey Mark” meme, I’m vaguely acquainted with Mark from LUG days)
- tgp 4y ago
  The other day I found out that not only ZFS and Btrfs have a dedup feature, but also XFS (which I use on my home partition) supports a feature called reflinks aka block sharing. So I threw duperemove at ~/.virtualenvs and ~/.cache/bazel and freed up a good 15GB of disk space. Good stuff.
MithrilTuxedo 4y ago
This sounds like it could be bolted onto/into rsync on the server side to present filesystems larger than the server can actually store.
maxyurk 4y ago
https://en.m.wikipedia.org/wiki/Content-addressable_storage https://en.m.wikipedia.org/wiki/Content-addressable_storage
- groestl 4y ago
  Also relevant in the context of partial file deduplication (to get robustness wrt inserts and deletions): https://en.wikipedia.org/wiki/Rolling_hash https://en.wikipedia.org/wiki/Rolling_hash
anyfoo 4y ago
Oh sweet. I've definitely used sqlite (in a zsh script) for file deduplication. Very simple, mostly just rows consisting of paths and file content hash. But partial file deduplication is something else...
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
fbdab103 4y ago
Does anyone else have any other unorthodox use cases? I love SQLite, and am always happy to ram this square peg into a round hole.
- aarchi 4y ago
  PostgreSQL is Turing-complete, as proven by implementations of a cyclic tag system[0] and Turing machine[1]. The Mandelbrot set[2] and travelling-salesman problem[3] have also been implemented in it. Transact-SQL is also Turing-complete, as proven by a Brainfuck implementation[4]. With that, you can theoretically compute anything :). [0]: https://wiki.postgresql.org/wiki/Cyclic_Tag_System https://wiki.postgresql.org/wiki/Cyclic_Tag_System [1]: https://blog.coelho.net/database/2013/08/17/turing-sql-1.html https://blog.coelho.net/database/2013/08/17/turing-sql-1.htm... [2]: https://wiki.postgresql.org/wiki/Mandelbrot_set https://wiki.postgresql.org/wiki/Mandelbrot_set [3]: https://web.archive.org/web/20201111224603/http://assets.en.oreilly.com/1/event/27/High%20Performance%20SQL%20with%20PostgreSQL%20Presentation.pdf https://web.archive.org/web/20201111224603/http://assets.en.... [4]: https://stackoverflow.com/questions/900055/is-sql-or-even-tsql-turing-complete/34847489#34847489 https://stackoverflow.com/questions/900055/is-sql-or-even-ts...
  - robertlagrant 4y ago
    Does any of the above apply to SQLite?
- yawaramin 4y ago
  Coincidentally I just wrote this about using SQLite for managing i18n strings: https://dev.to/yawaramin/translations-without-the-tears-1n9k https://dev.to/yawaramin/translations-without-the-tears-1n9k
tripleo1 4y ago
Slightly off topic -> there was a project I seen on gh that claimed to support system administration using a relational tables. Something like everything in SQL. I thought it might be a cool idea.
- coppsilgold 4y ago
  osquery <https://github.com/osquery/osquery https://github.com/osquery/osquery>
  - tripleo1 4y ago
    Looks familiar. Highly appreciated.
leetbulb 4y ago
"There is no file path, because macOS lets you look up files by id. The hash values are cryptographic hashes truncated to 64 bits and reinterpreted as integers." Is the author implying that APFS or HFS uses this method to calculate the file ID? I am unable to find any information regarding this. From what I understand, w.r.t APFS, the file ID is a combination of the inode OID and genID.
- cerved 4y ago
  I assumed they are taking about inodes
- jonhohle 4y ago
  The hash values are part of his implementation and don’t have anything to do with APFS or HFS. The sentence after your quote says, “I picked the BLAKE3 hash function…” indicating it was something he chose, not a FS implementation detail.
  - leetbulb 4y ago
    That's what I was leaning towards. The wording is weird to me. Thanks.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
megous 4y ago
I used a similar trick to stuff 14 Linux distros onto a 6 GiB sized SD card image using btrfs filesystem. The distros share a lot of common files, so this works well to save space. (btrfs also supports the data block sharing CoW feature as APFS) https://xnux.eu/p-boot-demo/ https://xnux.eu/p-boot-demo/ I had to write a custom tool to do it efficiently during image creation.
- tarasglek 4y ago
  This was neat. Found the important bits in https://megous.com/git/pinephone-multi-boot/tree/tools/main.c#n169 https://megous.com/git/pinephone-multi-boot/tree/tools/main....
- itsthecourier 4y ago
  You, savage. Kudos.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
knagy 4y ago
Reminds me of this article from Riot Games about how they deliver patches: https://technology.riotgames.com/news/supercharging-data-delivery-new-league-patcher https://technology.riotgames.com/news/supercharging-data-del...
- GOATS- 4y ago
  Thanks for the link, that was a fun read.
- blaz0 4y ago
  I appreciate seeing this article that I wrote 4 years ago resurface :) Happy to answer any questions about it or the tech behind it. We also use a SQLite database, and in a manner similar to the OP’s article. We use it to track which content-defined chunks are at which offsets in which files on disk. We deduplicate those chunks on the CDN so you have to download less data when updating the game, but on disk we need to recreate the files as originally uploaded because that’s how games load them. We’ve expanded the use of this tech quite a bit since the article. For example, my team has started using the patcher internally to manage our 2-million-file vendored code and binaries repo, as a replacement for Git LFS of sorts. Cloning the repo from scratch became 10 times faster, and so did switching branches, especially for remote devs, since the files are served through CloudFront. Some of the more interesting work we’ve done recently has been optimizing the core content-defined chunking algorithm. The GearHash rolling hash algorithm from FastCDC is decently fast, but we were able to improve chunking speeds by 5x with vectorized implementations and using AES-NI instructions in a clever way to avoid the GearHash table lookups. It can do 11GB/sec per core now using AVX-512 instructions. Another thing we did recently was build a tool for repacking Unreal Engine Pak files so game assets maintain a consistent layout across versions, similar to what garaetjjte mentioned in a comment above about UnityFS files. This reduced the disk I/O needed to update VALORANT by over 90% and cut update times by half for players. The combination of content-defined chunking with tooling to make game data files friendlier to differential patching can make a huge difference.
skerit 4y ago
I don't get it. How does this SQLite database interact with the AFS volume?
zubairq 4y ago
This is quite amazing. I actually built something even more crazy with Sqlite where I broke up the files into parts and then made hashes of the parts, so in total it would use less space than the sum of the files' space. i used this for a similarity engine where I would try to see how similar different files were

343 points by ics 4y ago

That's neat! In retrospect, did the database make the problem more tractable from an algorithmic or practical standpoint, or was it mostly just using the tools you're familiar with? If I were to approach the same problem, I likely would have tried to keep all the data in RAM and serialize it out to a file for incremental runs.

qsort 4y ago
IME a database does not algorithmically improve anything, but it does make at your disposal some very efficient data structures that are definitely nontrivial to replicate. This is especially the case if you have to work with datasets that don't fit in memory! An added bonus is that you can use SQL.
- zerkten 4y ago
  A friend works for a financial company that gets applicants from .NET and Linux backgrounds. He observed that the .NET devs almost always thought about solving these problems with a database and SQL that the Linux people would solve with pipelines of commands. Both were valid in his environment since they were both a big Oracle and SQL Server shop while having high Linux usage as well. I must ask him if PowerShell has changed things at all.
theamk 4y ago
Definitely the latter. There are many of other products which implement "keep track of partial file contents to deduplicate", for example casync borg and restic. None of them use sqlite for metadata storage, which kinda makes sense - in author's schema, each block uses (fileid, block_index, hash) tuple, which is 2/3 wasted space, as fileids are mostly same (if files are big) and block_index is always increasing. A custom data structure would be both faster and more efficient.
nonethewiser 4y ago
I have used SQLite to deduplicate csvs when it couldn’t all fit into memory. I imagine Dask would be a better tool for this job but I wasn’t familiar with it at the time and it did allow me to deduplicate using Python and no external dependencies.

hultner 4y ago

Very cool and probably a fun exercise, but I would probably put the data on a ZFS volume with dedupe instead, which from reading this implementation is pretty much the same thing to my layman’s perspective. I could also add compression to the same dataset.

traceroute66 4y ago
> I would probably put the data on a ZFS volume with dedupe instead But doesn't ZFS dedupe eat RAM for breakfast ? Double-digit GB RAM per TB data IIRC ?
- giantrobot 4y ago
  It's usually along the lines of 1GB per TB. Some factors can affect that number but I've found it pretty accurate. Note that's 1GB of RAM that ends up wired to ZFS and never usable by the rest of the system.
- redundantly 4y ago
  The DDT (deduplication table) used to require ~1GB RAM for every 1TB data written over the life of the file system. Deleting data from the file system wouldn't remove the dedupe references, you'd have to recreate the pool entirely and start over with a new DDT. However, there are now special devices that can be used stored to store DDT. Typically this is done with two SSDs configured as a mirrored vdev for the DDT metadata. This reduces the overhead on memory, but does cost some performance and still has the same limitation that the DDT size can only be reduced by re-creating the pool.
  - hultner 4y ago
    Yes this is what I do, and I wouldn't have dedupe for all datasets, for instance in this case he wanted dedupe on his games, so I would just enable it for that dataset. But I do run disk-wide compression with no problems and have done so on all my datasets for many years now, and it's been a tremendous space saver. Especially on machines where I have a lot of VMs/containers, it's not unusual for me to have a compression ratio of 2 on these with good old lz4, it will be interesting to see what damage zstd will do once I start experimenting with that.

evmar 4y ago

Coincidentally, I just saw that the "USENIX Test of Time Award" for 2023 was an analysis for deduplication purposes of real-world files. I found Figure 1 particularly interesting in that partial deduplication didn't save much over whole-file based deduplication in practice. https://www.usenix.org/conferences/test-of-time-awards https://www.usenix.org/conferences/test-of-time-awards

emmelaich 4y ago
You'd expect that in general, but for developers updating content the partial would often win.

mastax 4y ago

That's really cool. There are a bunch of tools that will let you symlink or hard link deduplicate files, but being able to do block-level dedupes simply by leaning on the filesystem is nice. It sometimes feels like games are made to thwart this type of thing. They often use packfiles, basically filesystems within files optimized to look up assets quickly. Also perhaps they allowed optimized data layout from when consoles had slow spinning hard drives. The upshot is that a tiny patch inserting a line of code in a script may offset hundreds of megabytes of other data in the packfiles, causing the block hashes to no longer match up. Do any filesystems model inserts in some way? I'm pretty sure Steam updates can handle situations like that. I frequently see updates which download a tiny amount (kilobytes) but write a huge amount to disk (gigabytes), and I can't think of any other cause. (Assuming developers aren't using hilariously un-compressed assets).

mikepurvis 4y ago
Yes, Steamworks encourages developers to align assets to certain boundaries to help ensure efficient patching later, see the documentation here: https://partner.steamgames.com/doc/sdk/uploading https://partner.steamgames.com/doc/sdk/uploading
- mastax 4y ago
  Interesting, it looks like steam doesn't have any special handling for inserts/offsets and just handles 1MB blocks. The extra disk IO probably just comes from rewriting large files: > Next - to update the pack file on a client device, SteamPipe builds the new version alongside the old version. When all new files are built, it then "commits" the update by deleting old files and moving the new files in. What this means is that to update a 25 GB pack file, SteamPipe will always build a new, 25GB file. If the update only requires 10 bytes of changes to that file, SteamPipe will need to copy almost the entire 25GB from the old file to the new file. Depending on the client storage hardware, this can be a very slow process.
  - bob1029 4y ago
    I think SQLite + WAL could be a compelling solution for game patching scenario. There are already SQLite clustering technologies that utilize WAL frame shipping in order to replicate data. I don't see why a steam patch couldn't just be a range of WAL frames to be applied to a client's database.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

rurban 4y ago

Using proper a hash table with a bloom filter would save you the useless pass through a b-tree though. Much faster and much less memory

pornel 4y ago
With a b-tree you can do even better. Instead of hashing entire files ahead of time, you can avoid hashing more than the minimum prefix required to conclude a file is unique, by building the b-tree lazily. I don't think it can be done with sqlite though, as it requires hashing the files while traversing the b-tree. https://lib.rs/crates/dupe-krill#readme-nerding-out-about-the-fast-deduplication-algorithm https://lib.rs/crates/dupe-krill#readme-nerding-out-about-th...

JackSlateur 4y ago

Looks like https://github.com/markfasheh/duperemove https://github.com/markfasheh/duperemove

philsnow 4y ago
Oh, hey Mark (Not (purely) a reference to the “oh hey Mark” meme, I’m vaguely acquainted with Mark from LUG days)
tgp 4y ago
The other day I found out that not only ZFS and Btrfs have a dedup feature, but also XFS (which I use on my home partition) supports a feature called reflinks aka block sharing. So I threw duperemove at ~/.virtualenvs and ~/.cache/bazel and freed up a good 15GB of disk space. Good stuff.

MithrilTuxedo 4y ago

This sounds like it could be bolted onto/into rsync on the server side to present filesystems larger than the server can actually store.

maxyurk 4y ago

https://en.m.wikipedia.org/wiki/Content-addressable_storage https://en.m.wikipedia.org/wiki/Content-addressable_storage

groestl 4y ago
Also relevant in the context of partial file deduplication (to get robustness wrt inserts and deletions): https://en.wikipedia.org/wiki/Rolling_hash https://en.wikipedia.org/wiki/Rolling_hash

anyfoo 4y ago

Oh sweet. I've definitely used sqlite (in a zsh script) for file deduplication. Very simple, mostly just rows consisting of paths and file content hash. But partial file deduplication is something else...

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

fbdab103 4y ago

Does anyone else have any other unorthodox use cases? I love SQLite, and am always happy to ram this square peg into a round hole.

aarchi 4y ago
PostgreSQL is Turing-complete, as proven by implementations of a cyclic tag system[0] and Turing machine[1]. The Mandelbrot set[2] and travelling-salesman problem[3] have also been implemented in it. Transact-SQL is also Turing-complete, as proven by a Brainfuck implementation[4]. With that, you can theoretically compute anything :). [0]: https://wiki.postgresql.org/wiki/Cyclic_Tag_System https://wiki.postgresql.org/wiki/Cyclic_Tag_System [1]: https://blog.coelho.net/database/2013/08/17/turing-sql-1.html https://blog.coelho.net/database/2013/08/17/turing-sql-1.htm... [2]: https://wiki.postgresql.org/wiki/Mandelbrot_set https://wiki.postgresql.org/wiki/Mandelbrot_set [3]: https://web.archive.org/web/20201111224603/http://assets.en.oreilly.com/1/event/27/High%20Performance%20SQL%20with%20PostgreSQL%20Presentation.pdf https://web.archive.org/web/20201111224603/http://assets.en.... [4]: https://stackoverflow.com/questions/900055/is-sql-or-even-tsql-turing-complete/34847489#34847489 https://stackoverflow.com/questions/900055/is-sql-or-even-ts...
- robertlagrant 4y ago
  Does any of the above apply to SQLite?
yawaramin 4y ago
Coincidentally I just wrote this about using SQLite for managing i18n strings: https://dev.to/yawaramin/translations-without-the-tears-1n9k https://dev.to/yawaramin/translations-without-the-tears-1n9k

tripleo1 4y ago

Slightly off topic -> there was a project I seen on gh that claimed to support system administration using a relational tables. Something like everything in SQL. I thought it might be a cool idea.

coppsilgold 4y ago
osquery <https://github.com/osquery/osquery https://github.com/osquery/osquery>
- tripleo1 4y ago
  Looks familiar. Highly appreciated.

leetbulb 4y ago

"There is no file path, because macOS lets you look up files by id. The hash values are cryptographic hashes truncated to 64 bits and reinterpreted as integers." Is the author implying that APFS or HFS uses this method to calculate the file ID? I am unable to find any information regarding this. From what I understand, w.r.t APFS, the file ID is a combination of the inode OID and genID.

cerved 4y ago
I assumed they are taking about inodes
jonhohle 4y ago
The hash values are part of his implementation and don’t have anything to do with APFS or HFS. The sentence after your quote says, “I picked the BLAKE3 hash function…” indicating it was something he chose, not a FS implementation detail.
- leetbulb 4y ago
  That's what I was leaning towards. The wording is weird to me. Thanks.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

megous 4y ago

I used a similar trick to stuff 14 Linux distros onto a 6 GiB sized SD card image using btrfs filesystem. The distros share a lot of common files, so this works well to save space. (btrfs also supports the data block sharing CoW feature as APFS) https://xnux.eu/p-boot-demo/ https://xnux.eu/p-boot-demo/ I had to write a custom tool to do it efficiently during image creation.

tarasglek 4y ago
This was neat. Found the important bits in https://megous.com/git/pinephone-multi-boot/tree/tools/main.c#n169 https://megous.com/git/pinephone-multi-boot/tree/tools/main....
itsthecourier 4y ago
You, savage. Kudos.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

knagy 4y ago

Reminds me of this article from Riot Games about how they deliver patches: https://technology.riotgames.com/news/supercharging-data-delivery-new-league-patcher https://technology.riotgames.com/news/supercharging-data-del...

GOATS- 4y ago
Thanks for the link, that was a fun read.
blaz0 4y ago
I appreciate seeing this article that I wrote 4 years ago resurface :) Happy to answer any questions about it or the tech behind it. We also use a SQLite database, and in a manner similar to the OP’s article. We use it to track which content-defined chunks are at which offsets in which files on disk. We deduplicate those chunks on the CDN so you have to download less data when updating the game, but on disk we need to recreate the files as originally uploaded because that’s how games load them. We’ve expanded the use of this tech quite a bit since the article. For example, my team has started using the patcher internally to manage our 2-million-file vendored code and binaries repo, as a replacement for Git LFS of sorts. Cloning the repo from scratch became 10 times faster, and so did switching branches, especially for remote devs, since the files are served through CloudFront. Some of the more interesting work we’ve done recently has been optimizing the core content-defined chunking algorithm. The GearHash rolling hash algorithm from FastCDC is decently fast, but we were able to improve chunking speeds by 5x with vectorized implementations and using AES-NI instructions in a clever way to avoid the GearHash table lookups. It can do 11GB/sec per core now using AVX-512 instructions. Another thing we did recently was build a tool for repacking Unreal Engine Pak files so game assets maintain a consistent layout across versions, similar to what garaetjjte mentioned in a comment above about UnityFS files. This reduced the disk I/O needed to update VALORANT by over 90% and cut update times by half for players. The combination of content-defined chunking with tooling to make game data files friendlier to differential patching can make a huge difference.

skerit 4y ago

I don't get it. How does this SQLite database interact with the AFS volume?

zubairq 4y ago

This is quite amazing. I actually built something even more crazy with Sqlite where I broke up the files into parts and then made hashes of the parts, so in total it would use less space than the sum of the files' space. i used this for a similarity engine where I would try to see how similar different files were