4 ms·
People often use sort | uniq when they don't want to load a bunch of lines into memory. That's why it's slow. It uses files and allocates very little memory by
by f311a 1y ago
People often use sort | uniq when they don't want to load a bunch of lines into memory. That's why it's slow. It uses files and allocates very little memory by default. The pros? You can sort hundreds of gigabytes of data.
This Rust implementation uses hashmap, if you have a lot of unique values, you will need a lot of RAM.
- noamteyssier 1y agoYeah definitely, it's always a trade-off. I think in many cases where I use it especially the number of unique values is actually not crazy high (much less than the required RAM) and the number of lines is crazy high. So in those settings I think it's absolutely worth it