4 ms·
How about this. For each row, compute an 8 bit hash and write to a file whose name is the hash value. Now you have 256 files that you can dedupe in memory (and
by imichael 4y ago
How about this. For each row, compute an 8 bit hash and write to a file whose name is the hash value. Now you have 256 files that you can dedupe in memory (and if not, use a 10 bit hash or whatever).
- nullbytesmatter 4y agoI have tried something similar, writing each entry to a trie on the filesystem and store each duplicate row on the filesystem. The problem was this created a heck of a mess and was taking too long, but it seems like it's feasible.