3 ms·
I created "unic" a number of years ago because I had need to get the unique lines from a giant file without losing the order they initially appeared. It achieve
by donatj 1y ago
I created "unic" a number of years ago because I had need to get the unique lines from a giant file without losing the order they initially appeared. It achieves this using a Cuckoo Filter so it's pretty dang quick about it, faster than sorting a large file in memory for sure.
https://github.com/donatj/unic https://github.com/donatj/unic
- deleted 1y ago[deleted]
- noamteyssier 1y agoNice tool!
- noamteyssier 1y agoI've actually added a benchmark for this specific task and added `unic` to it. It may not be the most fair comparison because with these random fastqs I'm generating the vast majority of the input is unique so it could be overloading the cuckoo filter.