With a b-tree you can do even better. Instead of hashing entire files ahead of time, you can avoid hashing more than the minimum prefix required to conclude a f

With a b-tree you can do even better. Instead of hashing entire files ahead of time, you can avoid hashing more than the minimum prefix required to conclude a file is unique, by building the b-tree lazily. I don't think it can be done with sqlite though, as it requires hashing the files while traversing the b-tree. https://lib.rs/crates/dupe-krill#readme-nerding-out-about-the-fast-deduplication-algorithm https://lib.rs/crates/dupe-krill#readme-nerding-out-about-th...