A-ha! I knew I had one more. How many times have you wanted to dedup a (text) file, but definitely didn't have enough memory to perform the task? I found this

A-ha! I knew I had one more. How many times have you wanted to dedup a (text) file, but definitely didn't have enough memory to perform the task? I found this one day when I had to dedup a set of .ndjson.gz files which totaled a cumulative 312 GBs. Utilizing the bloomfilter option, I was able to dedup the records without any large investment on my part. Anyways, runiq[1], "[an] efficient way to filter duplicate lines from input, à la uniq". It provides several ways to filter of which I almost always default to utilizing the bloomfilter implementation (`-f bloom`). --- [1] https://github.com/whitfin/runiq https://github.com/whitfin/runiq [2] https://whitfin.io/filtering-unique-logs-using-rust/ https://whitfin.io/filtering-unique-logs-using-rust/