2 ms·
Using proper a hash table with a bloom filter would save you the useless pass through a b-tree though. Much faster and much less memory
by rurban 4y ago
Using proper a hash table with a bloom filter would save you the useless pass through a b-tree though. Much faster and much less memory
- pornel 4y agoWith a b-tree you can do even better. Instead of hashing entire files ahead of time, you can avoid hashing more than the minimum prefix required to conclude a file is unique, by building the b-tree lazily. I don't think it can be done with sqlite though, as it requires hashing the files while traversing the b-tree. https://lib.rs/crates/dupe-krill#readme-nerding-out-about-the-fast-deduplication-algorithm https://lib.rs/crates/dupe-krill#readme-nerding-out-about-th...